Nov 03 15:05:23.747017 ip-10-0-134-23 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 15:05:23.747030 ip-10-0-134-23 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 15:05:23.747041 ip-10-0-134-23 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 15:05:23.747372 ip-10-0-134-23 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 15:05:33.959628 ip-10-0-134-23 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 15:05:33.959643 ip-10-0-134-23 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ace703fbc1b241818800a5140debe76f --
Nov 03 15:07:22.603304 ip-10-0-134-23 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 15:07:22.989569 ip-10-0-134-23 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 15:07:22.989569 ip-10-0-134-23 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 15:07:22.989569 ip-10-0-134-23 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 15:07:22.989569 ip-10-0-134-23 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 15:07:22.993107 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.992997    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:22.993107 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993106    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:22.993107 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993110    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993113    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993116    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993119    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993121    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993124    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993128    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993131    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993133    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993136    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993138    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993141    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993143    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993146    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993148    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993151    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993159    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993162    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993165    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:22.993199 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993167    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993170    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993172    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993175    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993178    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993180    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993182    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993185    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993187    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993190    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993192    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993195    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993197    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993201    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993204    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993206    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993209    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993212    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993214    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993217    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:22.993628 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993220    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993222    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993225    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993227    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993230    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993232    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993235    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993237    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993240    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993242    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993245    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993248    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993256    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993258    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993261    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993264    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993266    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993269    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993271    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:22.994113 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993274    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993276    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993279    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993281    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993284    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993286    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993289    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993292    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993294    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993297    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993299    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993302    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993306    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993310    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993313    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993316    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993319    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993322    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993324    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:22.994820 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993326    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993329    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993332    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993335    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993339    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993343    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993346    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993348    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993351    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993353    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993356    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993359    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993361    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993364    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993366    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993369    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993371    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993374    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993376    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993380    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:22.995465 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993382    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993385    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993387    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993390    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993392    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993394    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.993397    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995402    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995417    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995420    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995424    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995428    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995431    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995434    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995436    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995440    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995442    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995445    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995450    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995453    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:22.995937 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995928    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995933    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995936    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995940    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995943    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995946    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995949    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995951    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995955    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995957    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995980    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995984    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995987    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995989    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995992    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995995    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.995997    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996000    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996002    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996004    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:22.996444 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996008    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996010    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996013    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996016    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996018    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996021    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996024    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996026    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996029    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996031    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996034    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996036    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996039    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996041    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996045    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996048    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996050    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996053    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996056    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996059    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:22.996911 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996061    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996064    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996069    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996072    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996074    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996077    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996080    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996082    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996085    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996088    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996091    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996098    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996102    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996105    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996107    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996110    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996113    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996117    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996121    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:22.997446 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996124    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996128    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996133    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996135    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996138    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996140    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996143    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996147    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996150    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996153    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996156    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996158    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996161    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996164    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996166    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996169    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996171    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996174    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996176    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996179    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:22.997898 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996181    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996184    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996186    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996189    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996192    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996194    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996196    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996199    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996201    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996204    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996206    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996209    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996211    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996213    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996743    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996748    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996751    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996754    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996756    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:22.998387 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996760    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996763    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996765    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996768    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996772    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996774    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996777    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996779    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996782    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996784    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996786    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996789    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996791    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996795    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996799    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996802    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996804    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996807    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996809    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:22.998837 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996812    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996814    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996816    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996819    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996821    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996824    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996826    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996829    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996831    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996834    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996836    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996839    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996841    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996844    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996847    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996849    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996852    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996855    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996858    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996860    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:22.999331 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996863    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996865    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996868    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996870    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996875    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996879    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996882    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996885    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996887    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996890    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996892    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996895    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996897    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996900    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996902    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996904    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996907    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996910    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996912    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996915    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:22.999892 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996917    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996920    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996923    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996926    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996928    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996931    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996934    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996936    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996939    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996941    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996944    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996947    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996949    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996952    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996954    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996957    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996960    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996963    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996979    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:23.000384 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996982    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996985    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996987    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996990    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996992    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996995    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.996997    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997000    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997002    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997005    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997007    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997010    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997012    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997015    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997018    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997020    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997023    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997026    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997028    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997031    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:23.000809 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997034    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997036    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.997039    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998057    2573 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998068    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998075    2573 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998080    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998085    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998088    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998092    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998096    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998099    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998102    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998106    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998109    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998112    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998115    2573 flags.go:64] FLAG: --cgroup-root=""
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998118    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998121    2573 flags.go:64] FLAG: --client-ca-file=""
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998124    2573 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998127    2573 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998130    2573 flags.go:64] FLAG: --cluster-domain=""
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998133    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998136    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 15:07:23.001285 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998139    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998142    2573 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998146    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998149    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998152    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998158    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998161    2573 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998164    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998167    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998170    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998173    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998177    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998180    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998183    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998187    2573 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998190    2573 flags.go:64] FLAG: --enable-server="true"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998193    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998198    2573 flags.go:64] FLAG: --event-burst="100"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998201    2573 flags.go:64] FLAG: --event-qps="50"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998204    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998207    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998210    2573 flags.go:64] FLAG: --eviction-hard=""
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998214    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998217    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998220    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 15:07:23.001837 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998223    2573 flags.go:64] FLAG: --eviction-soft=""
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998226    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998229    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998232    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998235    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998237    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998240    2573 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998243    2573 flags.go:64] FLAG: --feature-gates=""
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998247    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998250    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998254    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998257    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998261    2573 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998264    2573 flags.go:64] FLAG: --help="false"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998267    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998270    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998273    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998276    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998279    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998283    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998285    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998292    2573 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998295    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 15:07:23.002490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998298    2573 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998301    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998304    2573 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998307    2573 flags.go:64] FLAG: --kube-reserved=""
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998310    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998313    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998316    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998319    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998321    2573 flags.go:64] FLAG: --lock-file=""
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998324    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998327    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998330    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998340    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998343    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998346    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998349    2573 flags.go:64] FLAG: --logging-format="text"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998352    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998355    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998358    2573 flags.go:64] FLAG: --manifest-url=""
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998361    2573 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998365    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998370    2573 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998374    2573 flags.go:64] FLAG: --max-pods="110"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998377    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998379    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 15:07:23.003045 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998382    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998385    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998388    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998391    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998394    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998401    2573 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998404    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998408    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998411    2573 flags.go:64] FLAG: --pod-cidr=""
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998414    2573 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998417    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998419    2573 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998422    2573 flags.go:64] FLAG: --port="10250"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998425    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998428    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02a26c39a727b0b5a"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998431    2573 flags.go:64] FLAG: --qos-reserved=""
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998433    2573 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998436    2573 flags.go:64] FLAG: --register-node="true"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998439    2573 flags.go:64] FLAG: --register-with-taints=""
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998443    2573 flags.go:64] FLAG: --registry-burst="10"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998446    2573 flags.go:64] FLAG: --registry-qps="5"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998449    2573 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998452    2573 flags.go:64] FLAG: --reserved-memory=""
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998456    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998459    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998462    2573 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 15:07:23.004351 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998465    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998467    2573 flags.go:64] FLAG: --runonce="false"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998471    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998474    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998477    2573 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998480    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998483    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998486    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998489    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998492    2573 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998495    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998498    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998501    2573 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998504    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998509    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998513    2573 flags.go:64] FLAG: --system-cgroups=""
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998516    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998522    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998524    2573 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998527    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998531    2573 flags.go:64] FLAG: --tls-min-version=""
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998533    2573 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998536    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998539    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998541    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 15:07:23.004985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998544    2573 flags.go:64] FLAG: --v="2"
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998549    2573 flags.go:64] FLAG: --version="false"
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998554    2573 flags.go:64] FLAG: --vmodule=""
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998557    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.998561    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998694    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998697    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998700    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998703    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998705    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998708    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998710    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998713    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998715    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998718    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998720    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998723    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998725    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998727    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998730    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998733    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:23.005811 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998735    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998740    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998742    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998745    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998747    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998750    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998752    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998755    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998757    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998760    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998762    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998765    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998767    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998770    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998772    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998775    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998778    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998780    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998782    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998785    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:23.006716 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998788    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998790    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998792    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998795    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998797    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998800    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998802    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998805    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998807    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998810    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998812    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998814    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998817    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998820    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998823    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998826    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998829    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998831    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998834    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998836    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:23.007615 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998839    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998842    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998844    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998847    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998851    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998855    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998859    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998861    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998864    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998866    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998869    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998871    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998874    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998876    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998878    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998881    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998883    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998886    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998888    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:23.008255 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998890    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998893    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998896    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998900    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998903    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998906    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998909    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998912    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998915    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998917    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998920    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998923    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998925    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998928    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998930    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998933    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998935    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998938    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998940    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998942    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:23.008815 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998945    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998947    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998950    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998952    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998954    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998958    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998960    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998963    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998980    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998982    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.998985    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:22.999013    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999056    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999060    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999063    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999066    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:23.009685 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999068    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999071    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999074    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999079    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999082    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999085    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999088    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999090    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999093    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999095    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999097    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999100    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999103    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999107    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999110    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999113    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999116    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999118    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999121    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:23.010412 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999123    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999126    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999129    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999132    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999135    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999137    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999139    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999142    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999144    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999147    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999149    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999153    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999157    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999160    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999163    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999166    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999171    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999173    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999176    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:23.010996 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999179    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999181    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999184    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999187    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999190    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999192    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999195    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999197    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999199    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999202    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999204    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999206    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999209    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999212    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999214    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999218    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999220    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999223    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999225    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999228    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:23.011541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999230    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999233    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999235    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999238    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999240    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999242    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999245    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999248    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999250    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999254    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999256    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999259    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999261    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999264    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999267    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999269    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999272    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999274    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999277    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999279    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:23.012098 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999282    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999284    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999286    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999289    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999291    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999294    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999296    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999300    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999302    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999305    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999307    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999310    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999312    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999315    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999317    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999320    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999322    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999324    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999327    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999329    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:23.012581 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999331    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999336    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999338    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:22.999342    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.007777    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.007797    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.008571    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.008584    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008673    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008681    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008686    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008693    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008700    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008705    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008709    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008713    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008718    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008722    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008726    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:23.013075 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008730    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008735    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008739    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008744    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008749    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008753    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008758    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008762    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008766    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008770    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008775    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008779    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008784    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008788    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008792    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008796    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008801    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008805    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008811    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:23.013672 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008815    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008820    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008824    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008828    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008832    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008836    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008840    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008844    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008848    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008852    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008856    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008861    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008865    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008869    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008873    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008877    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008881    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008885    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008889    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008893    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:23.014467 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008897    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008902    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008906    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008910    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008914    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008918    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008922    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008927    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008931    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008935    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008939    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008944    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008950    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008954    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008959    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008963    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008987    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008992    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.008996    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009000    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:23.014949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009004    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009009    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009013    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009017    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009021    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009025    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009029    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009033    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009037    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009041    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009045    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009050    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009054    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009058    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009062    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009066    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009070    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009074    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009078    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:23.015651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009082    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009086    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009091    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009095    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009099    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009104    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009110    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009115    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009119    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009123    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009127    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009132    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009136    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009140    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009144    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009148    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:23.016115 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009155    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.009198    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009429    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009437    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009443    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009448    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009452    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009456    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009461    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009465    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009469    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009473    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009477    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009481    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009486    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009490    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:23.016541 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009494    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009498    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009502    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009507    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009511    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009515    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009519    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009524    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009529    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009533    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009537    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009541    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009545    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009549    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009553    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009557    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009561    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009565    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009570    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:23.016919 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009575    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009579    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009583    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009590    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009595    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009600    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009604    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009609    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009613    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009617    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009621    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009625    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009631    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009638    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009642    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009647    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009651    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009655    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009660    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:23.017443 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009664    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009668    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009674    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009678    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009681    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009685    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009689    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009693    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009696    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009700    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009704    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009708    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009712    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009716    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009721    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009725    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009729    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009733    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009737    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009741    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:23.017875 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009745    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009749    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009753    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009758    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009762    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009766    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009770    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009774    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009779    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009783    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009786    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009793    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009797    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009801    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009806    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009811    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009815    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009819    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009823    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009828    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:23.018556 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009832    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009836    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009839    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009844    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009848    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009853    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009857    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009861    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009865    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009869    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009873    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009877    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009881    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009885    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.009923    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.009998    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:07:23.019508 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010007    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010014    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010019    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010023    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010028    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010032    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010037    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010043    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010048    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010054    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010059    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010063    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010068    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010073    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010077    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010081    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010085    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010090    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010094    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:07:23.020200 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010098    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010103    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010107    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010111    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010115    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010119    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010123    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010127    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010131    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010135    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010139    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010144    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010148    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010152    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010156    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010160    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010164    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010168    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010173    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010177    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:07:23.020787 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010182    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010187    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010191    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010195    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010199    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010203    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010208    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010212    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010217    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010221    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010225    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010229    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010233    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010238    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010243    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010247    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010251    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010255    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010260    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010264    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:07:23.021276 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010268    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010273    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010277    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010281    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010285    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010289    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010293    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010297    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010301    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010306    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010310    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010314    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010320    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010324    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010328    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010331    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010335    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010340    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010344    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:07:23.021746 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010349    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010353    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010357    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010361    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010365    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010369    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010373    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010377    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010381    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010385    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010389    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010393    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010397    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010401    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010405    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010409    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010413    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010417    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010421    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010425    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:07:23.022201 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010429    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010433    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010437    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010441    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010445    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010455    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:23.010459    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.010546    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.015241    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.016308    2573 server.go:1014] "Starting client certificate rotation"
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.016406    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.016441    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.018768    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 15:07:23.022679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.020853    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 15:07:23.032790 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.032769    2573 log.go:25] "Validated CRI v1 runtime API"
Nov 03 15:07:23.036644 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.036629    2573 log.go:25] "Validated CRI v1 image API"
Nov 03 15:07:23.037403 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.037381    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 15:07:23.040027 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.040008    2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 baae2c0b-2d9d-45f9-9871-c6a90e3c9eb1:/dev/nvme0n1p3 fb40d821-1043-4a78-975e-5c1f1fdf5d70:/dev/nvme0n1p4]
Nov 03 15:07:23.040093 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.040027    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 15:07:23.045119 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.045011    2573 manager.go:217] Machine: {Timestamp:2026-11-03 15:07:23.043576897 +0000 UTC m=+0.346239105 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100339 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d41e76c2d2310f138a148db54fe0 SystemUUID:ec20d41e-76c2-d231-0f13-8a148db54fe0 BootID:ace703fb-c1b2-4181-8800-a5140debe76f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:3c:c8:36:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:3c:c8:36:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:f7:19:0b:42:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 15:07:23.045119 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.045117    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 15:07:23.045245 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.045234    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 15:07:23.048451 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.048430    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 15:07:23.048577 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.048453    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-23.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 15:07:23.048617 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.048580    2573 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 15:07:23.048617 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.048588    2573 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 15:07:23.048666 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.048622    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 15:07:23.048666 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.048640    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 15:07:23.049870 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.049860    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 15:07:23.054702 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.054690    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 15:07:23.054811 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.054802    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 15:07:23.055566 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.055557    2573 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 15:07:23.055595 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.055571    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 15:07:23.055595 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.055581    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 15:07:23.055595 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.055590    2573 kubelet.go:405] "Adding apiserver pod source"
Nov 03 15:07:23.055672 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.055599    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 15:07:23.055949 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.055938    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 15:07:23.058193 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.058165    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 15:07:23.065774 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.065760    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 15:07:23.066657 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066645    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 15:07:23.066691 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066676    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 15:07:23.066851 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066841    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 15:07:23.066882 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066858    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 15:07:23.066882 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066864    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 15:07:23.066882 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066870    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 15:07:23.066882 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066876    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 15:07:23.066994 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.066891    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 15:07:23.067590 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.067581    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 15:07:23.067625 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.067591    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 15:07:23.067625 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.067599    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 15:07:23.067625 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.067605    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 15:07:23.067625 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.067618    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 15:07:23.067625 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.067626    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 15:07:23.068370 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.068360    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 15:07:23.068408 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.068373    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 15:07:23.072258 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.072244    2573 server.go:1263] "Started kubelet"
Nov 03 15:07:23.072336 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.072307    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 15:07:23.072521 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.072353    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 15:07:23.072570 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.072552    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 15:07:23.073107 ip-10-0-134-23 systemd[1]: Started Kubernetes Kubelet.
Nov 03 15:07:23.074159 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.074142    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 15:07:23.075284 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.075267    2573 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 15:07:23.078444 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.078424    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 15:07:23.080868 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.080851    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 15:07:23.081432 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.081415    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 15:07:23.081540 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.081519    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found"
Nov 03 15:07:23.082325 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.082289    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 15:07:23.082487 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.082289    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 15:07:23.082487 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.082340    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 15:07:23.082849 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.082821    2573 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 15:07:23.082849 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.082837    2573 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 15:07:23.083331 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083314    2573 factory.go:55] Registering systemd factory
Nov 03 15:07:23.083396 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083354    2573 factory.go:223] Registration of the systemd container factory successfully
Nov 03 15:07:23.083651 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083636    2573 factory.go:153] Registering CRI-O factory
Nov 03 15:07:23.083651 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083652    2573 factory.go:223] Registration of the crio container factory successfully
Nov 03 15:07:23.083765 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083706    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 15:07:23.083765 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083725    2573 factory.go:103] Registering Raw factory
Nov 03 15:07:23.083765 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.083749    2573 manager.go:1199] Started watching for new ooms in manager
Nov 03 15:07:23.084903 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.084844    2573 manager.go:319] Starting recovery of all containers
Nov 03 15:07:23.090781 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.090733    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 15:07:23.096564 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.096547    2573 manager.go:324] Recovery completed
Nov 03 15:07:23.097789 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.097767    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 15:07:23.100769 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.100756    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.102425 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.102389    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.102485 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.102451    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.102485 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.102479    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.103036 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103018    2573 cpu_manager.go:226] "Starting" policy="none"
Nov 03 15:07:23.103036 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103035    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 15:07:23.103164 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103054    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 15:07:23.103164 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103026    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.103732 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103717    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.103774 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103749    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.103774 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.103766    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.105252 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.105239    2573 policy_none.go:50] "Start"
Nov 03 15:07:23.105313 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.105255    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 15:07:23.105313 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.105270    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 15:07:23.107381 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.107370    2573 policy_none.go:44] "Start"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.152519    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.156828    2573 manager.go:341] "Starting Device Plugin manager"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.156861    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.156872    2573 server.go:87] "Starting device plugin registration server"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.157158    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.157169    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.157358    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.157455    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.157462    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.157861    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.157905    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-23.us-west-1.compute.internal\" not found"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.159628    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.159650    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.159667    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 15:07:23.164730 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.159695    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 15:07:23.224561 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.222370    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-23.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 15:07:23.224561 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.222371    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-23.us-west-1.compute.internal.18e4916ba262d9d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-23.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-23.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:23.072215506 +0000 UTC m=+0.374877709,LastTimestamp:2026-11-03 15:07:23.072215506 +0000 UTC m=+0.374877709,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-23.us-west-1.compute.internal,}"
Nov 03 15:07:23.258072 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.257988    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.258849 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.258832    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.258949 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.258868    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.258949 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.258883    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.258949 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.258916    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.260483 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.260465    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"]
Nov 03 15:07:23.260543 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.260521    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.261198 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.261182    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.261274 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.261215    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.261274 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.261226    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.263504 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.263492    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.263668 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.263649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.263706 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.263685    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.264236 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.264221    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.264289 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.264249    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.264289 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.264260    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.264562 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.264550    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.264627 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.264577    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.264627 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.264587    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.266753 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.266739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.266815 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.266764    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.267423 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.267405    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.267492 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.267437    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.267492 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.267452    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.271126 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.271108    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-23.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 15:07:23.284521 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.284492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.284611 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.284533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.284611 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.284563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc7811c328f411b1828c6a10a8039cb-config\") pod \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"6cc7811c328f411b1828c6a10a8039cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.288486 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.288469    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.290561 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.290545    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.308025 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.308009    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 15:07:23.308095 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.308063    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrhjv"
Nov 03 15:07:23.319031 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.319001    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 15:07:23.319172 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.319100    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-23.us-west-1.compute.internal.18e4916ba42fe900  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-23.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-23.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-23.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:23.102431488 +0000 UTC m=+0.405093694,LastTimestamp:2026-11-03 15:07:23.102431488 +0000 UTC m=+0.405093694,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-23.us-west-1.compute.internal,}"
Nov 03 15:07:23.384292 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.384240    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:07:23.384483 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.384247    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:07:23.384553 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.384536    2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-23.us-west-1.compute.internal\" not found" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.385515 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385496    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:07:23.385561 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc7811c328f411b1828c6a10a8039cb-config\") pod \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"6cc7811c328f411b1828c6a10a8039cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.385561 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.385631 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.385631 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385615    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.385631 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385615    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc7811c328f411b1828c6a10a8039cb-config\") pod \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"6cc7811c328f411b1828c6a10a8039cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.385736 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.385635    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.527715 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.527636    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-23.us-west-1.compute.internal\" not found" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.585003 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.584961    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:07:23.586105 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.586090    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:07:23.586184 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.586119    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:07:23.586184 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.586135    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:07:23.586184 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.586162    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.593252 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.593238    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.596401 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.596388    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.645050 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.645015    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.645050 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:23.645045    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-23.us-west-1.compute.internal\": node \"ip-10-0-134-23.us-west-1.compute.internal\" not found"
Nov 03 15:07:23.683020 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.682988    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.761722 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.761687    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 15:07:23.762801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.762785    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:07:23.821755 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:23.821683    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 15:07:24.016015 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.015983    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 15:07:24.016576 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.016168    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 15:07:24.016576 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.016174    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 15:07:24.016576 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.016142    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2ce9ff9192244e67a5502d625692119-459a7f00d10b34a9.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-23.us-west-1.compute.internal.18e4916ba43050ce\": read tcp 10.0.134.23:51276->3.151.164.251:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-23.us-west-1.compute.internal.18e4916ba43050ce  default   4664 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-23.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-23.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-23.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 15:07:23 +0000 UTC,LastTimestamp:2026-11-03 15:07:23.261220216 +0000 UTC m=+0.563882420,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-23.us-west-1.compute.internal,}"
Nov 03 15:07:24.056120 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.056094    2573 apiserver.go:52] "Watching apiserver"
Nov 03 15:07:24.081952 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.081890    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 15:07:24.314584 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.314538    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:02:23 +0000 UTC" deadline="2028-08-20 09:46:43.737062262 +0000 UTC"
Nov 03 15:07:24.314584 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.314577    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15738h39m19.422489307s"
Nov 03 15:07:24.354532 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.354457    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 15:07:24.354752 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.354734    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-ltnrw","kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74","openshift-image-registry/node-ca-j87d4","openshift-multus/multus-fgwcf","openshift-ovn-kubernetes/ovnkube-node-kzzmg","openshift-network-operator/iptables-alerter-zz979","kube-system/global-pull-secret-syncer-xnsmp","openshift-cluster-node-tuning-operator/tuned-8rfqr","openshift-network-diagnostics/network-check-target-8lrk7","kube-system/konnectivity-agent-pq4lr","openshift-dns/node-resolver-9gcrg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-vdz4p"]
Nov 03 15:07:24.357258 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.357243    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.359403 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.359382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.361469 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.361453    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.363447 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.363431    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.365610 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.365593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.367925 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.367899    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.368414 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.368386    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:24.370637 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.370616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.370711 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.370680    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:24.372697 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.372678    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.374855 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.374841    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:24.374922 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.374890    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:24.377143 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.377125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.379389 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.379374    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.381637 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.381618    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.383679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.383662    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 15:07:24.390363 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afcc4bb-db5b-431a-bdb9-580af2eccd18-agent-certs\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.390448 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-ovn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390448 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.390448 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-lib-modules\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.390448 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390412    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-node-log\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390571 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-modprobe-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.390571 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-host\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.390571 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390535    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-iptables-alerter-script\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.390654 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.390654 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.390654 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ff874b-b837-480b-8adb-266dcaac4f07-tmp-dir\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.390654 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390766 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-env-overrides\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390766 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-cnibin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.390766 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-device-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.390766 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-slash\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390766 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390743    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtchn\" (UniqueName: \"kubernetes.io/projected/bc79121f-b8ff-433d-962e-e6c5d02a65f8-kube-api-access-gtchn\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.390897 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-netns\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.390897 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390787    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-hostroot\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.390897 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390810    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.390897 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-etc-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390897 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.390897 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysconfig\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc79121f-b8ff-433d-962e-e6c5d02a65f8-serviceca\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-etc-kubernetes\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.390960    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj98j\" (UniqueName: \"kubernetes.io/projected/55ee9f5d-10b0-4ae3-946b-5295148f2429-kube-api-access-zj98j\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01ff874b-b837-480b-8adb-266dcaac4f07-hosts-file\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391030    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-systemd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-log-socket\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391082 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-bin\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391086    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-dbus\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-system-cni-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-kubernetes\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391177    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-systemd\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-conf-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-sys\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-multus\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391264 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-daemon-config\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7dn\" (UniqueName: \"kubernetes.io/projected/d116bb71-10c3-40c9-ac2d-50a79bc72946-kube-api-access-wg7dn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-run\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-host-slash\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-socket-dir-parent\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391366    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gb27\" (UniqueName: \"kubernetes.io/projected/01ff874b-b837-480b-8adb-266dcaac4f07-kube-api-access-7gb27\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-netns\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391434    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-k8s-cni-cncf-io\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.391528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-cnibin\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-cni-binary-copy\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-systemd-units\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391679    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jrl\" (UniqueName: \"kubernetes.io/projected/f94b8f15-cb42-44c9-bbdc-6c609dec30db-kube-api-access-69jrl\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc79121f-b8ff-433d-962e-e6c5d02a65f8-host\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-bin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-script-lib\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxt7\" (UniqueName: \"kubernetes.io/projected/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-kube-api-access-blxt7\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.391794 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-tmp\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-kubelet-config\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-multus-certs\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqx28\" (UniqueName: \"kubernetes.io/projected/c6171645-b2a3-4bf8-8258-a5aef22950e7-kube-api-access-jqx28\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391872    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afcc4bb-db5b-431a-bdb9-580af2eccd18-konnectivity-ca\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-kubelet\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-netd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-var-lib-kubelet\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.391985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-config\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwzx\" (UniqueName: \"kubernetes.io/projected/58ea0b3a-4929-431a-8cfe-410028747797-kube-api-access-tdwzx\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-conf\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-tuned\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6zd\" (UniqueName: \"kubernetes.io/projected/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-kube-api-access-zq6zd\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.392242 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-var-lib-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.392679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392158    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovn-node-metrics-cert\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.392679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392171    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-system-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.392679 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.392201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-kubelet\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.410568 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.410545    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdt56\""
Nov 03 15:07:24.410648 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.410633    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.410648 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.410637    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.411724 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.411710    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 15:07:24.411953 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.411942    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wckw\""
Nov 03 15:07:24.412025 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.412013    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 15:07:24.414030 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.414017    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.414065 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.414048    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jpxkd\""
Nov 03 15:07:24.414115 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.414104    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 15:07:24.414152 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.414128    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.416084 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.416070    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.416130 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.416088    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 15:07:24.416170 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.416132    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cpt6\""
Nov 03 15:07:24.416211 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.416175    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.418327 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.418309    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 15:07:24.418458 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.418442    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.418544 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.418533    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 15:07:24.457549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.457524    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.457655 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.457602    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 15:07:24.457655 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.457618    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 15:07:24.460173 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.460159    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 15:07:24.460749 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.460731    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.460749 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.460732    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsch2\""
Nov 03 15:07:24.460873 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.460848    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.460927 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.460910    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.462222 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462197    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.462222 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462217    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 15:07:24.462222 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462204    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 15:07:24.462420 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462260    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsh2r\""
Nov 03 15:07:24.462420 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462275    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 15:07:24.462420 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462297    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 15:07:24.462420 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462320    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 15:07:24.462420 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.462323    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 15:07:24.466021 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.465999    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjtnz\""
Nov 03 15:07:24.466098 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.466089    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cblvv\""
Nov 03 15:07:24.466516 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.466484    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfz9p\""
Nov 03 15:07:24.466548 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.466520    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 15:07:24.466580 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.466568    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vgz6r"
Nov 03 15:07:24.492394 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.492491 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-config\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.492491 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.492491 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.492655 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492483    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.492655 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwzx\" (UniqueName: \"kubernetes.io/projected/58ea0b3a-4929-431a-8cfe-410028747797-kube-api-access-tdwzx\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.492655 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.492563    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:24.492655 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-conf\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.492655 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.492633    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:07:24.992611741 +0000 UTC m=+2.295273936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-tuned\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492683    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-conf\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6zd\" (UniqueName: \"kubernetes.io/projected/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-kube-api-access-zq6zd\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-var-lib-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovn-node-metrics-cert\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-system-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-kubelet\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492804    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afcc4bb-db5b-431a-bdb9-580af2eccd18-agent-certs\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.492921 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-ovn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492957    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-var-lib-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.492959    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493008    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-lib-modules\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493050    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-system-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-node-log\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-modprobe-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493092    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-config\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-host\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-iptables-alerter-script\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-host\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ff874b-b837-480b-8adb-266dcaac4f07-tmp-dir\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-env-overrides\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-modprobe-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.493382 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-cnibin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493288    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-lib-modules\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493315    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-node-log\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493332    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-cnibin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-device-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-slash\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493387    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtchn\" (UniqueName: \"kubernetes.io/projected/bc79121f-b8ff-433d-962e-e6c5d02a65f8-kube-api-access-gtchn\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493107    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-kubelet\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-netns\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-hostroot\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493497    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-etc-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493550    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-netns\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysconfig\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493590    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc79121f-b8ff-433d-962e-e6c5d02a65f8-serviceca\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.494283 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493600    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-etc-kubernetes\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj98j\" (UniqueName: \"kubernetes.io/projected/55ee9f5d-10b0-4ae3-946b-5295148f2429-kube-api-access-zj98j\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01ff874b-b837-480b-8adb-266dcaac4f07-hosts-file\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493670    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ff874b-b837-480b-8adb-266dcaac4f07-tmp-dir\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-systemd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493672    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-slash\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-log-socket\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493715    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysconfig\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-bin\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493740    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493751    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-hostroot\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-dbus\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493788    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-etc-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-system-cni-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-kubernetes\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.495197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-systemd\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493920    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-iptables-alerter-script\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493924    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-conf-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493952    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-conf-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-sys\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494010    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-multus\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-daemon-config\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.494072    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7dn\" (UniqueName: \"kubernetes.io/projected/d116bb71-10c3-40c9-ac2d-50a79bc72946-kube-api-access-wg7dn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494148    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-run\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494191    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc79121f-b8ff-433d-962e-e6c5d02a65f8-serviceca\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.494198    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret podName:1e3decdb-1a61-4186-9f1d-c41cac5bb6da nodeName:}" failed. No retries permitted until 2026-11-03 15:07:24.994180901 +0000 UTC m=+2.296843108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret") pod "global-pull-secret-syncer-xnsmp" (UID: "1e3decdb-1a61-4186-9f1d-c41cac5bb6da") : object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-host-slash\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-system-cni-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494241    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-etc-kubernetes\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496083 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-run\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494264    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494285    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-host-slash\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494287    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-systemd\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494309    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-log-socket\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494321    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-bin\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.493640    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-device-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494342    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-socket-dir-parent\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494382    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-systemd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-socket-dir-parent\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494393    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01ff874b-b837-480b-8adb-266dcaac4f07-hosts-file\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb27\" (UniqueName: \"kubernetes.io/projected/01ff874b-b837-480b-8adb-266dcaac4f07-kube-api-access-7gb27\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494421    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-dbus\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-netns\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494469    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-k8s-cni-cncf-io\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.496734 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494099    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-env-overrides\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-cnibin\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494603    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-ovn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494637    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494659    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-k8s-cni-cncf-io\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-cni-binary-copy\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494706    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-netns\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-systemd-units\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jrl\" (UniqueName: \"kubernetes.io/projected/f94b8f15-cb42-44c9-bbdc-6c609dec30db-kube-api-access-69jrl\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc79121f-b8ff-433d-962e-e6c5d02a65f8-host\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-bin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494808    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-script-lib\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494651    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-kubernetes\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxt7\" (UniqueName: \"kubernetes.io/projected/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-kube-api-access-blxt7\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-tmp\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494897    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-kubelet-config\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494926    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-daemon-config\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-multus-certs\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494986    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-multus-certs\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.494986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqx28\" (UniqueName: \"kubernetes.io/projected/c6171645-b2a3-4bf8-8258-a5aef22950e7-kube-api-access-jqx28\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afcc4bb-db5b-431a-bdb9-580af2eccd18-konnectivity-ca\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-kubelet\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-netd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-var-lib-kubelet\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495100    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-systemd-units\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495179    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495187    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-sys\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495227    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-multus\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.497632 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495283    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495305    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc79121f-b8ff-433d-962e-e6c5d02a65f8-host\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495317    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495353    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-bin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495357    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-cni-binary-copy\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495414    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-kubelet-config\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495422    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-kubelet\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495466    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-cnibin\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495495    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-netd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495513    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-var-lib-kubelet\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.495784    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-script-lib\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.496104    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afcc4bb-db5b-431a-bdb9-580af2eccd18-konnectivity-ca\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.496815    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovn-node-metrics-cert\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.496852    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afcc4bb-db5b-431a-bdb9-580af2eccd18-agent-certs\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.497016    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-tuned\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.498088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.497187    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-tmp\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.555576 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.555534    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 15:07:24.555576 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.555571    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 15:07:24.555576 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.555585    2573 projected.go:196] Error preparing data for projected volume kube-api-access-dzkcj for pod openshift-network-diagnostics/network-check-target-8lrk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:24.555812 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.555665    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj podName:7351fb71-99f2-4358-bd82-895d4ffd76d9 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:25.055640882 +0000 UTC m=+2.358303098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dzkcj" (UniqueName: "kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj") pod "network-check-target-8lrk7" (UID: "7351fb71-99f2-4358-bd82-895d4ffd76d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:24.556503 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.556357    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6zd\" (UniqueName: \"kubernetes.io/projected/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-kube-api-access-zq6zd\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.557690 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.557665    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7dn\" (UniqueName: \"kubernetes.io/projected/d116bb71-10c3-40c9-ac2d-50a79bc72946-kube-api-access-wg7dn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.557993 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.557959    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jrl\" (UniqueName: \"kubernetes.io/projected/f94b8f15-cb42-44c9-bbdc-6c609dec30db-kube-api-access-69jrl\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.558121 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.558101    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxt7\" (UniqueName: \"kubernetes.io/projected/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-kube-api-access-blxt7\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.558233 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.558215    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqx28\" (UniqueName: \"kubernetes.io/projected/c6171645-b2a3-4bf8-8258-a5aef22950e7-kube-api-access-jqx28\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.558571 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.558557    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj98j\" (UniqueName: \"kubernetes.io/projected/55ee9f5d-10b0-4ae3-946b-5295148f2429-kube-api-access-zj98j\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.562985 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.562950    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb27\" (UniqueName: \"kubernetes.io/projected/01ff874b-b837-480b-8adb-266dcaac4f07-kube-api-access-7gb27\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.564821 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.564797    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwzx\" (UniqueName: \"kubernetes.io/projected/58ea0b3a-4929-431a-8cfe-410028747797-kube-api-access-tdwzx\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.569719 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.569698    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtchn\" (UniqueName: \"kubernetes.io/projected/bc79121f-b8ff-433d-962e-e6c5d02a65f8-kube-api-access-gtchn\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.690555 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.690522    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:07:24.708249 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.708220    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:24.727399 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.727377    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:07:24.744581 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.744552    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:07:24.771530 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.771501    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:24.774574 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.774554    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fgwcf"
Nov 03 15:07:24.791528 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.791505    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:07:24.795023 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:24.794995    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ee9f5d_10b0_4ae3_946b_5295148f2429.slice/crio-6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6 WatchSource:0}: Error finding container 6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6: Status 404 returned error can't find the container with id 6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6
Nov 03 15:07:24.795924 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:24.795867    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd116bb71_10c3_40c9_ac2d_50a79bc72946.slice/crio-d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f WatchSource:0}: Error finding container d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f: Status 404 returned error can't find the container with id d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f
Nov 03 15:07:24.803182 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.803164    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:07:24.803253 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:24.803228    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6171645_b2a3_4bf8_8258_a5aef22950e7.slice/crio-7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c WatchSource:0}: Error finding container 7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c: Status 404 returned error can't find the container with id 7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c
Nov 03 15:07:24.813765 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.813747    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:07:24.824195 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:24.824174    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ff874b_b837_480b_8adb_266dcaac4f07.slice/crio-c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e WatchSource:0}: Error finding container c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e: Status 404 returned error can't find the container with id c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e
Nov 03 15:07:24.829523 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.829506    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:07:24.839024 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:24.838995    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ea0b3a_4929_431a_8cfe_410028747797.slice/crio-989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974 WatchSource:0}: Error finding container 989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974: Status 404 returned error can't find the container with id 989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974
Nov 03 15:07:24.937607 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.937567    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:07:24.947256 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.947178    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:07:24.997727 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.997685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:24.997918 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:24.997738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:24.997918 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.997849    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:24.997918 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.997855    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:24.997918 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.997907    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret podName:1e3decdb-1a61-4186-9f1d-c41cac5bb6da nodeName:}" failed. No retries permitted until 2026-11-03 15:07:25.997890268 +0000 UTC m=+3.300552477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret") pod "global-pull-secret-syncer-xnsmp" (UID: "1e3decdb-1a61-4186-9f1d-c41cac5bb6da") : object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:24.998124 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:24.997923    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:07:25.997916562 +0000 UTC m=+3.300578758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:25.099081 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.099046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:25.099795 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:25.099220    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 15:07:25.099795 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:25.099238    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 15:07:25.099795 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:25.099248    2573 projected.go:196] Error preparing data for projected volume kube-api-access-dzkcj for pod openshift-network-diagnostics/network-check-target-8lrk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:25.099795 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:25.099300    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj podName:7351fb71-99f2-4358-bd82-895d4ffd76d9 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:26.099286586 +0000 UTC m=+3.401948780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzkcj" (UniqueName: "kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj") pod "network-check-target-8lrk7" (UID: "7351fb71-99f2-4358-bd82-895d4ffd76d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:25.164153 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.164094    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"6cc7811c328f411b1828c6a10a8039cb","Type":"ContainerStarted","Data":"b30b6a09896114a41acb1a647b64fc71d56733ea3a173911af2879e3b82800e6"}
Nov 03 15:07:25.165135 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.165106    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerStarted","Data":"989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974"}
Nov 03 15:07:25.166225 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.166203    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9gcrg" event={"ID":"01ff874b-b837-480b-8adb-266dcaac4f07","Type":"ContainerStarted","Data":"c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e"}
Nov 03 15:07:25.167303 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.167274    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74" event={"ID":"c6171645-b2a3-4bf8-8258-a5aef22950e7","Type":"ContainerStarted","Data":"7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c"}
Nov 03 15:07:25.168429 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.168400    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fgwcf" event={"ID":"55ee9f5d-10b0-4ae3-946b-5295148f2429","Type":"ContainerStarted","Data":"6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6"}
Nov 03 15:07:25.169530 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.169509    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f"}
Nov 03 15:07:25.170511 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.170489    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"daa380c87733219694504b47fb5e04ca","Type":"ContainerStarted","Data":"3467489b98251c4ccb2dfce8777edd2995d6697c463ee5bba8794ee7a3ec07ef"}
Nov 03 15:07:25.470717 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.470649    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:24 +0000 UTC" deadline="2028-08-07 18:38:38.131539726 +0000 UTC"
Nov 03 15:07:25.470717 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.470706    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15435h31m12.660835927s"
Nov 03 15:07:25.555116 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:25.555074    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94b8f15_cb42_44c9_bbdc_6c609dec30db.slice/crio-e75f727c4f438ad23acb9bbef32d5f46f98b97c136d26b521919cd0b6888a5c7 WatchSource:0}: Error finding container e75f727c4f438ad23acb9bbef32d5f46f98b97c136d26b521919cd0b6888a5c7: Status 404 returned error can't find the container with id e75f727c4f438ad23acb9bbef32d5f46f98b97c136d26b521919cd0b6888a5c7
Nov 03 15:07:25.613651 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:25.613613    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14d1e13_22f0_4ab1_b82c_e1694965e8d8.slice/crio-7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290 WatchSource:0}: Error finding container 7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290: Status 404 returned error can't find the container with id 7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290
Nov 03 15:07:25.614119 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:25.614097    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc79121f_b8ff_433d_962e_e6c5d02a65f8.slice/crio-df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737 WatchSource:0}: Error finding container df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737: Status 404 returned error can't find the container with id df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737
Nov 03 15:07:25.649454 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:25.649433    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:07:26.005271 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.005238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:26.005475 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.005308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:26.005475 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.005467    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:26.005620 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.005547    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:07:28.005523439 +0000 UTC m=+5.308185653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:26.005691 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.005622    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:26.005748 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.005701    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret podName:1e3decdb-1a61-4186-9f1d-c41cac5bb6da nodeName:}" failed. No retries permitted until 2026-11-03 15:07:28.005688001 +0000 UTC m=+5.308350210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret") pod "global-pull-secret-syncer-xnsmp" (UID: "1e3decdb-1a61-4186-9f1d-c41cac5bb6da") : object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:26.105885 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.105844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:26.106439 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.106030    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 15:07:26.106439 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.106051    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 15:07:26.106439 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.106065    2573 projected.go:196] Error preparing data for projected volume kube-api-access-dzkcj for pod openshift-network-diagnostics/network-check-target-8lrk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:26.106439 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.106127    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj podName:7351fb71-99f2-4358-bd82-895d4ffd76d9 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:28.106108354 +0000 UTC m=+5.408770562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzkcj" (UniqueName: "kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj") pod "network-check-target-8lrk7" (UID: "7351fb71-99f2-4358-bd82-895d4ffd76d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:26.160145 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.160108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:26.160340 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.160252    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:26.160406 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.160353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:26.160461 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.160428    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:26.160514 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.160494    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:26.160599 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:26.160576    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:26.174914 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.174877    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq4lr" event={"ID":"2afcc4bb-db5b-431a-bdb9-580af2eccd18","Type":"ContainerStarted","Data":"3fe45434e54cac550be0c55af949b7f67d6ffdd65f1a01dd915bc0d35ed2b879"}
Nov 03 15:07:26.176700 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.176656    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j87d4" event={"ID":"bc79121f-b8ff-433d-962e-e6c5d02a65f8","Type":"ContainerStarted","Data":"df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737"}
Nov 03 15:07:26.181969 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.181939    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zz979" event={"ID":"b14d1e13-22f0-4ab1-b82c-e1694965e8d8","Type":"ContainerStarted","Data":"7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290"}
Nov 03 15:07:26.184722 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.184693    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8rfqr" event={"ID":"f94b8f15-cb42-44c9-bbdc-6c609dec30db","Type":"ContainerStarted","Data":"e75f727c4f438ad23acb9bbef32d5f46f98b97c136d26b521919cd0b6888a5c7"}
Nov 03 15:07:26.471757 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.471685    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:24 +0000 UTC" deadline="2028-04-27 03:09:15.644606824 +0000 UTC"
Nov 03 15:07:26.471757 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:26.471755    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12972h1m49.172856284s"
Nov 03 15:07:28.022852 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:28.022804    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:28.023272 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:28.022887    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:28.023272 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.023020    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:28.023272 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.023125    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:07:32.023107962 +0000 UTC m=+9.325770158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:28.023272 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.023201    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:28.023272 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.023238    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret podName:1e3decdb-1a61-4186-9f1d-c41cac5bb6da nodeName:}" failed. No retries permitted until 2026-11-03 15:07:32.023226609 +0000 UTC m=+9.325888806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret") pod "global-pull-secret-syncer-xnsmp" (UID: "1e3decdb-1a61-4186-9f1d-c41cac5bb6da") : object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:28.124175 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:28.124124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:28.124384 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.124334    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 15:07:28.124384 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.124354    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 15:07:28.124384 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.124369    2573 projected.go:196] Error preparing data for projected volume kube-api-access-dzkcj for pod openshift-network-diagnostics/network-check-target-8lrk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:28.124585 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.124459    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj podName:7351fb71-99f2-4358-bd82-895d4ffd76d9 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:32.124402664 +0000 UTC m=+9.427064862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzkcj" (UniqueName: "kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj") pod "network-check-target-8lrk7" (UID: "7351fb71-99f2-4358-bd82-895d4ffd76d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:28.160341 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:28.160312    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:28.160498 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.160441    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:28.160565 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:28.160518    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:28.160617 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.160591    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:28.160671 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:28.160656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:28.160756 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:28.160737    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:30.160554 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.160166    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:30.160554 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.160448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:30.160554 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.160412    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:30.161134 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:30.160600    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:30.161134 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:30.161092    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:30.161233 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:30.161200    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:30.225853 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.225082    2573 generic.go:356] "Generic (PLEG): container finished" podID="daa380c87733219694504b47fb5e04ca" containerID="c988ead8bb5284ee601ce22ee10878ef6bcd50be2e27f5b598d52104135c5ee2" exitCode=0
Nov 03 15:07:30.225853 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.225432    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"daa380c87733219694504b47fb5e04ca","Type":"ContainerDied","Data":"c988ead8bb5284ee601ce22ee10878ef6bcd50be2e27f5b598d52104135c5ee2"}
Nov 03 15:07:30.230433 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.229905    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"6cc7811c328f411b1828c6a10a8039cb","Type":"ContainerStarted","Data":"3a0bb60eed91caa39ee80d4541fef0f36f03517f1b628eb482623635b6ea3647"}
Nov 03 15:07:30.240462 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.240328    2573 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="94c6febcca2330dde31dd74cf98684342efd9f1794e184c08c277d09e801d856" exitCode=0
Nov 03 15:07:30.240462 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:30.240373    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerDied","Data":"94c6febcca2330dde31dd74cf98684342efd9f1794e184c08c277d09e801d856"}
Nov 03 15:07:31.248923 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:31.248884    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"daa380c87733219694504b47fb5e04ca","Type":"ContainerStarted","Data":"499ca2a7909923ef00a925a8f00dbe6e82078353c3264992b1a8951b66460ceb"}
Nov 03 15:07:31.254916 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:31.254889    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq4lr" event={"ID":"2afcc4bb-db5b-431a-bdb9-580af2eccd18","Type":"ContainerStarted","Data":"1863438218f5396ed9aa15f722fa91977f727fd2e6045bc97d64d8890f0d714f"}
Nov 03 15:07:31.363232 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:31.363174    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal" podStartSLOduration=7.363154518 podStartE2EDuration="7.363154518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:30.603239034 +0000 UTC m=+7.905901250" watchObservedRunningTime="2026-11-03 15:07:31.363154518 +0000 UTC m=+8.665816732"
Nov 03 15:07:31.478537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:31.478478    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pq4lr" podStartSLOduration=3.569838141 podStartE2EDuration="8.478458515s" totalImagesPullingTime="4.908620374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:31.478207981 +0000 UTC m=+8.780870196" watchObservedRunningTime="2026-11-03 15:07:31.478458515 +0000 UTC m=+8.781120729"
Nov 03 15:07:31.478674 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:31.478637    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" podStartSLOduration=8.478629472 podStartE2EDuration="8.478629472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:31.364253048 +0000 UTC m=+8.666915289" watchObservedRunningTime="2026-11-03 15:07:31.478629472 +0000 UTC m=+8.781291687"
Nov 03 15:07:32.052409 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.052374    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:32.052573 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.052443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:32.052573 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.052558    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:32.052685 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.052601    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret podName:1e3decdb-1a61-4186-9f1d-c41cac5bb6da nodeName:}" failed. No retries permitted until 2026-11-03 15:07:40.052589185 +0000 UTC m=+17.355251375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret") pod "global-pull-secret-syncer-xnsmp" (UID: "1e3decdb-1a61-4186-9f1d-c41cac5bb6da") : object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:32.052685 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.052641    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:32.052685 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.052661    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:07:40.052654575 +0000 UTC m=+17.355316765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:32.153319 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.153281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:32.153477 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.153466    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 15:07:32.153512 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.153486    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 15:07:32.153512 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.153498    2573 projected.go:196] Error preparing data for projected volume kube-api-access-dzkcj for pod openshift-network-diagnostics/network-check-target-8lrk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:32.153573 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.153558    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj podName:7351fb71-99f2-4358-bd82-895d4ffd76d9 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:40.153540709 +0000 UTC m=+17.456202920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzkcj" (UniqueName: "kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj") pod "network-check-target-8lrk7" (UID: "7351fb71-99f2-4358-bd82-895d4ffd76d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:32.160753 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.160333    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:32.160753 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.160456    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:32.160753 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.160527    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:32.160753 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.160595    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:32.160753 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.160648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:32.160753 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:32.160722    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:32.259689 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.259599    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74" event={"ID":"c6171645-b2a3-4bf8-8258-a5aef22950e7","Type":"ContainerStarted","Data":"82941f1d3df7a0c9ada1c5e6b003a3db7ab10f82d30d51b87d2a012b2f6c8e53"}
Nov 03 15:07:32.263446 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:32.262924    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j87d4" event={"ID":"bc79121f-b8ff-433d-962e-e6c5d02a65f8","Type":"ContainerStarted","Data":"369b46b9f39c37f0d2bce640909e3fe25c74eee02dac3c9f5c18a55f799c9744"}
Nov 03 15:07:33.948590 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:33.948195    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 15:07:34.160910 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.160875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:34.161133 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:34.161026    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:34.161133 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.161100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:34.161247 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:34.161181    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:34.161247 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.161234    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:34.161319 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:34.161295    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:34.165670 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.165512    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T15:07:33.948218792Z","UUID":"e57b110d-10ba-4768-8aa0-ed62e2f304ed","Handler":null,"Name":"","Endpoint":""}
Nov 03 15:07:34.167664 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.167641    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 15:07:34.167756 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.167679    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 15:07:34.279869 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.279609    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74" event={"ID":"c6171645-b2a3-4bf8-8258-a5aef22950e7","Type":"ContainerStarted","Data":"5343edc490bf81e8f3d5792ac57498409db0ffa29d2dec5e6ba9ba003eb46f90"}
Nov 03 15:07:34.281882 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.281838    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fgwcf" event={"ID":"55ee9f5d-10b0-4ae3-946b-5295148f2429","Type":"ContainerStarted","Data":"d30aacea750ef21f3ca7479e4737bde5665f633ea8c0126344a53c904ff4100b"}
Nov 03 15:07:34.284933 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.284908    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9gcrg" event={"ID":"01ff874b-b837-480b-8adb-266dcaac4f07","Type":"ContainerStarted","Data":"2d5f215cec43b4ba4617fc7c5e54415a58ef7d4300e0b1b4dde7173f9b4c7d27"}
Nov 03 15:07:34.399803 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.399733    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j87d4" podStartSLOduration=5.363511423 podStartE2EDuration="11.399715078s" totalImagesPullingTime="6.036203655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:32.38515432 +0000 UTC m=+9.687816534" watchObservedRunningTime="2026-11-03 15:07:34.399715078 +0000 UTC m=+11.702377304"
Nov 03 15:07:34.513936 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.513835    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9gcrg" podStartSLOduration=1.61325381 podStartE2EDuration="10.513821264s" totalImagesPullingTime="8.900567454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:34.513770227 +0000 UTC m=+11.816432441" watchObservedRunningTime="2026-11-03 15:07:34.513821264 +0000 UTC m=+11.816483477"
Nov 03 15:07:34.514373 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:34.514339    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fgwcf" podStartSLOduration=1.720028379 podStartE2EDuration="10.514326135s" totalImagesPullingTime="8.794297756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:34.400398291 +0000 UTC m=+11.703060515" watchObservedRunningTime="2026-11-03 15:07:34.514326135 +0000 UTC m=+11.816988351"
Nov 03 15:07:35.291352 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:35.291309    2573 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="7289a25436bab7b6629770039b321d7aacabb7782188fcf8c5177473db1b64c6" exitCode=0
Nov 03 15:07:35.292409 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:35.292382    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerDied","Data":"7289a25436bab7b6629770039b321d7aacabb7782188fcf8c5177473db1b64c6"}
Nov 03 15:07:35.669529 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:35.669492    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:35.670732 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:35.670101    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:36.160446 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.160409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:36.160609 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.160485    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:36.160609 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.160509    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:36.160609 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:36.160590    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:36.160741 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:36.160649    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:36.160741 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:36.160690    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:36.295844 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.295806    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74" event={"ID":"c6171645-b2a3-4bf8-8258-a5aef22950e7","Type":"ContainerStarted","Data":"e110627498e7bc10875279229ca62e95d1386b8e2fda11e24ea287c06167666a"}
Nov 03 15:07:36.297355 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.297325    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zz979" event={"ID":"b14d1e13-22f0-4ab1-b82c-e1694965e8d8","Type":"ContainerStarted","Data":"da540367ea6e9255516b114c92665d64a3d75f38b022f915b59645ad5e35a4e9"}
Nov 03 15:07:36.297494 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.297414    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:36.298001 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.297984    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:07:36.423228 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.423140    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74" podStartSLOduration=2.184097967 podStartE2EDuration="12.423126591s" totalImagesPullingTime="10.239028624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:36.422923676 +0000 UTC m=+13.725585890" watchObservedRunningTime="2026-11-03 15:07:36.423126591 +0000 UTC m=+13.725788952"
Nov 03 15:07:36.651997 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:36.651928    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zz979" podStartSLOduration=4.089909041 podStartE2EDuration="12.651909282s" totalImagesPullingTime="8.562000241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:36.651621225 +0000 UTC m=+13.954283440" watchObservedRunningTime="2026-11-03 15:07:36.651909282 +0000 UTC m=+13.954571495"
Nov 03 15:07:37.301076 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:37.301033    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8rfqr" event={"ID":"f94b8f15-cb42-44c9-bbdc-6c609dec30db","Type":"ContainerStarted","Data":"3a3d9672e399c30b17ce58bf347c5077088bd37b797f88dc22467255f1c82a27"}
Nov 03 15:07:37.303769 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:37.303740    2573 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="148a028f65e5a5e2962d32cf690a994da39d6882071a1d8f2e6ef1312deba648" exitCode=0
Nov 03 15:07:37.303892 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:37.303819    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerDied","Data":"148a028f65e5a5e2962d32cf690a994da39d6882071a1d8f2e6ef1312deba648"}
Nov 03 15:07:37.416117 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:37.416061    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8rfqr" podStartSLOduration=3.386994626 podStartE2EDuration="14.416048513s" totalImagesPullingTime="11.029053887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:37.416005038 +0000 UTC m=+14.718667252" watchObservedRunningTime="2026-11-03 15:07:37.416048513 +0000 UTC m=+14.718710725"
Nov 03 15:07:38.160071 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:38.160037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:38.160258 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:38.160037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:38.160258 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:38.160173    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:38.160258 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:38.160185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:38.160375 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:38.160279    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:38.160375 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:38.160356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:39.313521 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:39.313287    2573 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="eff950eb491ace9ea2abe8db376b8057805a9fa93d7e893a0e83f41c480d665a" exitCode=0
Nov 03 15:07:39.314009 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:39.313553    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerDied","Data":"eff950eb491ace9ea2abe8db376b8057805a9fa93d7e893a0e83f41c480d665a"}
Nov 03 15:07:40.120991 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:40.120943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:40.121190 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:40.121012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:40.121190 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.121095    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:40.121190 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.121097    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:40.121190 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.121141    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret podName:1e3decdb-1a61-4186-9f1d-c41cac5bb6da nodeName:}" failed. No retries permitted until 2026-11-03 15:07:56.121129526 +0000 UTC m=+33.423791723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret") pod "global-pull-secret-syncer-xnsmp" (UID: "1e3decdb-1a61-4186-9f1d-c41cac5bb6da") : object "kube-system"/"original-pull-secret" not registered
Nov 03 15:07:40.121190 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.121165    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:07:56.121148126 +0000 UTC m=+33.423810321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 15:07:40.160135 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:40.160108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:40.160236 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:40.160187    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:40.160282 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:40.160272    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:40.160315 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.160286    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:40.160357 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.160340    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:40.160403 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.160352    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:40.222277 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:40.222243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:40.222417 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.222398    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 15:07:40.222456 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.222420    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 15:07:40.222456 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.222430    2573 projected.go:196] Error preparing data for projected volume kube-api-access-dzkcj for pod openshift-network-diagnostics/network-check-target-8lrk7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:40.222585 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:40.222473    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj podName:7351fb71-99f2-4358-bd82-895d4ffd76d9 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:56.222460962 +0000 UTC m=+33.525123167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzkcj" (UniqueName: "kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj") pod "network-check-target-8lrk7" (UID: "7351fb71-99f2-4358-bd82-895d4ffd76d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 15:07:41.321357 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.321122    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:07:41.321758 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.321730    2573 generic.go:356] "Generic (PLEG): container finished" podID="d116bb71-10c3-40c9-ac2d-50a79bc72946" containerID="c635da4725b69cb518362bb55969e9bc1026dda34d704a305b4f9d421860eb14" exitCode=1
Nov 03 15:07:41.321805 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.321786    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"24e9ca572134260f4d4ba56216256d3b9e9d91589a9796ae25d31d927110fc58"}
Nov 03 15:07:41.321874 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.321861    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"11c069e408f144e92e0b0a99a7650a60b794d79c75c75578e479ed9165148170"}
Nov 03 15:07:41.321925 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.321914    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"12c8c35cb3c4f596f99d2b4b9ea37c4f5641828af9324828121951849874d27b"}
Nov 03 15:07:41.322008 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.321963    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"aea86c4ba2b34721b6aff81e67b41f387011d5d7736767ef25f28bdaccd308e0"}
Nov 03 15:07:41.322059 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.322046    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerDied","Data":"c635da4725b69cb518362bb55969e9bc1026dda34d704a305b4f9d421860eb14"}
Nov 03 15:07:41.322109 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:41.322098    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"e5eb29c091683ef035cf4434daa130de002e19ac5ade539e1cbad7efc9501a7e"}
Nov 03 15:07:42.160408 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:42.160366    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:42.160559 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:42.160382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:42.160559 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:42.160488    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:42.160559 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:42.160486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:42.160559 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:42.160553    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:42.160730 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:42.160655    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:42.326865 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:42.326832    2573 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="bbf88af498c4fbbc07479c6944de5fb3977b64099a428c23de6a90d159064436" exitCode=0
Nov 03 15:07:42.327333 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:42.326891    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerDied","Data":"bbf88af498c4fbbc07479c6944de5fb3977b64099a428c23de6a90d159064436"}
Nov 03 15:07:43.332085 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:43.331998    2573 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="170bb9a849a8468475fa09475d938e11a1eb4e40f2deb49815dfff0a770552ff" exitCode=0
Nov 03 15:07:43.332668 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:43.332081    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerDied","Data":"170bb9a849a8468475fa09475d938e11a1eb4e40f2deb49815dfff0a770552ff"}
Nov 03 15:07:43.335024 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:43.335009    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:07:43.335337 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:43.335319    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"63c4839b6b320c863b2d99a39ae27ebeabc2a46aa2b0f95c226fa6040acaa2b3"}
Nov 03 15:07:44.160687 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:44.160648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:44.160849 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:44.160737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:44.160849 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:44.160803    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:44.160849 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:44.160818    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:44.160963 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:44.160871    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:44.160963 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:44.160926    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:44.340615 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:44.340584    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" event={"ID":"58ea0b3a-4929-431a-8cfe-410028747797","Type":"ContainerStarted","Data":"39fa588566bd2d85b0c52cd033e6a1d55b920a235757dcaad08dde4d8e801b09"}
Nov 03 15:07:44.458001 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:44.457899    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vdz4p" podStartSLOduration=7.849483682 podStartE2EDuration="20.4578862s" totalImagesPullingTime="12.608402518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:44.457617042 +0000 UTC m=+21.760279256" watchObservedRunningTime="2026-11-03 15:07:44.4578862 +0000 UTC m=+21.760548410"
Nov 03 15:07:45.347418 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:45.347315    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:07:45.347956 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:45.347746    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"a1b080278761a9b0616337d162a94ab17ad4c0540e45c3d3cfc7a796e26b8f27"}
Nov 03 15:07:45.348281 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:45.348262    2573 scope.go:122] "RemoveContainer" containerID="c635da4725b69cb518362bb55969e9bc1026dda34d704a305b4f9d421860eb14"
Nov 03 15:07:46.160158 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.160120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:46.160425 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.160120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:46.160425 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:46.160229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:46.160425 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.160120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:46.160425 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:46.160298    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:46.160425 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:46.160367    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:46.361825 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.361798    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:07:46.362239 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.362218    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" event={"ID":"d116bb71-10c3-40c9-ac2d-50a79bc72946","Type":"ContainerStarted","Data":"083ec37bc77e22551d21891a6b877d7ebaeb181c101e865702edcfe9e4319c7c"}
Nov 03 15:07:46.362319 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.362304    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:07:46.362552 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.362533    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:46.362624 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.362601    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:46.377216 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.377183    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:46.377358 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.377284    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:46.628795 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:46.628697    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg" podStartSLOduration=6.921577903 podStartE2EDuration="22.628683776s" totalImagesPullingTime="15.707105873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:46.578633977 +0000 UTC m=+23.881296189" watchObservedRunningTime="2026-11-03 15:07:46.628683776 +0000 UTC m=+23.931345988"
Nov 03 15:07:47.364778 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.364753    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:07:47.713537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.713499    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8lrk7"]
Nov 03 15:07:47.713702 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.713621    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:47.713749 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:47.713720    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:47.717019 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.716994    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ltnrw"]
Nov 03 15:07:47.717139 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.717106    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:47.717209 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:47.717191    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:47.717619 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.717598    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xnsmp"]
Nov 03 15:07:47.717711 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:47.717681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:47.717759 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:47.717744    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:48.366544 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:48.366516    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:07:49.160085 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:49.160055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:49.160085 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:49.160073    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:49.160274 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:49.160055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:49.160533 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:49.160504    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:49.160664 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:49.160644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:49.160783 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:49.160765    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:51.160639 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:51.160599    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:51.161129 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:51.160608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:51.161129 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:51.160702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:51.161129 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:51.160750    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:51.161129 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:51.160782    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:51.161129 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:51.160859    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:53.161225 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.161181    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:53.161684 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.161293    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xnsmp" podUID="1e3decdb-1a61-4186-9f1d-c41cac5bb6da"
Nov 03 15:07:53.161684 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.161311    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:53.161684 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.161371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:53.161684 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.161460    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8lrk7" podUID="7351fb71-99f2-4358-bd82-895d4ffd76d9"
Nov 03 15:07:53.161684 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.161512    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltnrw" podUID="2ede9faa-be51-4210-9fa6-1dc7ddbca38b"
Nov 03 15:07:53.481211 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.481032    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeReady"
Nov 03 15:07:53.481375 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.481306    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 15:07:53.636079 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.636046    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"]
Nov 03 15:07:53.639262 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.639240    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"]
Nov 03 15:07:53.639435 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.639415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.642696 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.642678    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.649490 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.649469    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"]
Nov 03 15:07:53.654275 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.654255    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qb946"]
Nov 03 15:07:53.654390 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.654373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.657724 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.657708    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lxnsf"]
Nov 03 15:07:53.657845 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.657822    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:07:53.661001 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.660982    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gh68l"]
Nov 03 15:07:53.661079 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.661038    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:07:53.664068 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.664051    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:53.664150 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.664099    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b6f864dcc-6c9p5"]
Nov 03 15:07:53.664202 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.664158    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:07:53.664272 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.664256    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:07:53.668080 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.668058    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-x66cq"]
Nov 03 15:07:53.668357 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.668177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.671625 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.671604    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"]
Nov 03 15:07:53.671746 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.671719    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.674799 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.674783    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-w7zh5"]
Nov 03 15:07:53.675098 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.675079    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:07:53.677860 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.677841    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:07:53.677956 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.677908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.686563 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.686544    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-644965599b-fltb2"]
Nov 03 15:07:53.689549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.689531    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"]
Nov 03 15:07:53.689653 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.689639    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.694779 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.694759    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-592hq"]
Nov 03 15:07:53.694915 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.694896    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.697755 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.697738    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:53.700062 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.700041    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64c988bcf7-rdfnp"]
Nov 03 15:07:53.702798 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.702781    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"]
Nov 03 15:07:53.702894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.702882    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.705614 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.705598    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6vv5d"]
Nov 03 15:07:53.705740 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.705722    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:07:53.708549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.708525    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kz2vz"]
Nov 03 15:07:53.710239 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.709298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.711158 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.711261 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711168    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.711261 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-config\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.711364 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/5e1df8e2-38c0-4e61-a117-f3b797ec963b-kube-api-access-ncmgr\") pod \"kas-connection-checker-685895c7c9-lxnsf\" (UID: \"5e1df8e2-38c0-4e61-a117-f3b797ec963b\") " pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:07:53.711513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8db8c20d-d878-475b-97d0-6cd01befd3ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.711588 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.711588 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3827824-759c-446a-b653-bed9ccf417c2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.711690 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-config\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.711739 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdjh\" (UniqueName: \"kubernetes.io/projected/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-kube-api-access-djdjh\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.711833 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711817    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpnz\" (UniqueName: \"kubernetes.io/projected/f3827824-759c-446a-b653-bed9ccf417c2-kube-api-access-2dpnz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.711915 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.711915 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp85d\" (UniqueName: \"kubernetes.io/projected/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-kube-api-access-pp85d\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.711915 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-stats-auth\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbgm\" (UniqueName: \"kubernetes.io/projected/8db8c20d-d878-475b-97d0-6cd01befd3ae-kube-api-access-jkbgm\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4e32724-e514-41cc-b175-59da7bfd7b8b-serving-cert\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.711998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2pq\" (UniqueName: \"kubernetes.io/projected/852144f2-7df8-4279-bf86-65c5351c8945-kube-api-access-2b2pq\") pod \"kas-connection-checker-685895c7c9-gh68l\" (UID: \"852144f2-7df8-4279-bf86-65c5351c8945\") " pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-trusted-ca\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhdj\" (UniqueName: \"kubernetes.io/projected/a4e32724-e514-41cc-b175-59da7bfd7b8b-kube-api-access-6hhdj\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bn2b\" (UniqueName: \"kubernetes.io/projected/f50eed36-43f3-4f6d-8aa0-46a148ca8177-kube-api-access-5bn2b\") pod \"kas-connection-checker-685895c7c9-qb946\" (UID: \"f50eed36-43f3-4f6d-8aa0-46a148ca8177\") " pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-serving-cert\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.712171 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3827824-759c-446a-b653-bed9ccf417c2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.712649 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvvj\" (UniqueName: \"kubernetes.io/projected/1df716cc-1298-4e45-bc66-f8d50e25353f-kube-api-access-phvvj\") pod \"volume-data-source-validator-58c9f5f46d-nn7mt\" (UID: \"1df716cc-1298-4e45-bc66-f8d50e25353f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:07:53.712649 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.712215    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-default-certificate\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.720637 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.720615    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 15:07:53.720776 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.720674    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.720776 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.720724    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 15:07:53.720992 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.720813    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.720992 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.720885    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.722763 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.722744    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.722862 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.722793    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 15:07:53.722862 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.722842    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 15:07:53.724577 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.724555    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"]
Nov 03 15:07:53.725467 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.725442    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"]
Nov 03 15:07:53.726140 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.726120    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"]
Nov 03 15:07:53.726953 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.726930    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"]
Nov 03 15:07:53.727603 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.727585    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qb946"]
Nov 03 15:07:53.736654 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.736629    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lxnsf"]
Nov 03 15:07:53.737197 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.737179    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-x66cq"]
Nov 03 15:07:53.742579 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.742559    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-w7zh5"]
Nov 03 15:07:53.743445 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.743428    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"]
Nov 03 15:07:53.744049 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.744028    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"]
Nov 03 15:07:53.744647 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.744634    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-592hq"]
Nov 03 15:07:53.771564 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.771531    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vv5d"]
Nov 03 15:07:53.774674 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.774649    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gh68l"]
Nov 03 15:07:53.778832 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.778803    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b6f864dcc-6c9p5"]
Nov 03 15:07:53.799667 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.799636    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644965599b-fltb2"]
Nov 03 15:07:53.807206 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.807181    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 15:07:53.808709 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.808691    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.808835 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.808719    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 15:07:53.808835 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.808784    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.808979 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.808958    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 15:07:53.809036 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809018    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.809146 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809062    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809233    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809253    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809264    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809274    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809266    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809307    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809374    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 15:07:53.809465 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809389    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809543    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809560    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809569    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809566    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809584    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809585    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809610    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809671    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809683    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809711    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809766    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809773    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809815    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809833    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809842    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809862    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809863    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809867    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.809876 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.809886    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 15:07:53.810628 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.810256    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.810628 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.810471    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 15:07:53.813814 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.813787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.813902 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.813825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-config\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.813902 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.813855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.813902 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.813896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-cert\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.814067 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.813921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8db8c20d-d878-475b-97d0-6cd01befd3ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.814067 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.813945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3827824-759c-446a-b653-bed9ccf417c2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.814067 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.813989    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.31394592 +0000 UTC m=+31.616608112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:07:53.814067 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814035    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.814284 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpnz\" (UniqueName: \"kubernetes.io/projected/f3827824-759c-446a-b653-bed9ccf417c2-kube-api-access-2dpnz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.814284 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.814284 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.814284 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-stats-auth\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.814284 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce6d3-5601-4133-b805-b378c9c45dd8-config-volume\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814295    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbgm\" (UniqueName: \"kubernetes.io/projected/8db8c20d-d878-475b-97d0-6cd01befd3ae-kube-api-access-jkbgm\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d078cc9-6850-4884-b8d2-b7e5b87f087e-serving-cert\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxn8v\" (UniqueName: \"kubernetes.io/projected/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-kube-api-access-mxn8v\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-trusted-ca\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.814513 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhdj\" (UniqueName: \"kubernetes.io/projected/a4e32724-e514-41cc-b175-59da7bfd7b8b-kube-api-access-6hhdj\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814514    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-config\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814522    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3827824-759c-446a-b653-bed9ccf417c2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ce6d3-5601-4133-b805-b378c9c45dd8-tmp-dir\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-service-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814743    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6krj\" (UniqueName: \"kubernetes.io/projected/273b804d-46fc-4201-b6d1-1029b44b301d-kube-api-access-r6krj\") pod \"network-check-source-6f78d59f99-qmkmp\" (UID: \"273b804d-46fc-4201-b6d1-1029b44b301d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvvj\" (UniqueName: \"kubernetes.io/projected/1df716cc-1298-4e45-bc66-f8d50e25353f-kube-api-access-phvvj\") pod \"volume-data-source-validator-58c9f5f46d-nn7mt\" (UID: \"1df716cc-1298-4e45-bc66-f8d50e25353f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:07:53.814801 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-default-certificate\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1612bb40-f413-40d9-89c2-284ddc106ca7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.814958    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.814927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2pq\" (UniqueName: \"kubernetes.io/projected/852144f2-7df8-4279-bf86-65c5351c8945-kube-api-access-2b2pq\") pod \"kas-connection-checker-685895c7c9-gh68l\" (UID: \"852144f2-7df8-4279-bf86-65c5351c8945\") " pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.815106    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.315091083 +0000 UTC m=+31.617753287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815171    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/5e1df8e2-38c0-4e61-a117-f3b797ec963b-kube-api-access-ncmgr\") pod \"kas-connection-checker-685895c7c9-lxnsf\" (UID: \"5e1df8e2-38c0-4e61-a117-f3b797ec963b\") " pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.815209    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:53.815235 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-config\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.815251    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.315240573 +0000 UTC m=+31.617902765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdjh\" (UniqueName: \"kubernetes.io/projected/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-kube-api-access-djdjh\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cplr8\" (UniqueName: \"kubernetes.io/projected/355ce6d3-5601-4133-b805-b378c9c45dd8-kube-api-access-cplr8\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp85d\" (UniqueName: \"kubernetes.io/projected/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-kube-api-access-pp85d\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-tmp\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-snapshots\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4e32724-e514-41cc-b175-59da7bfd7b8b-serving-cert\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bn2b\" (UniqueName: \"kubernetes.io/projected/f50eed36-43f3-4f6d-8aa0-46a148ca8177-kube-api-access-5bn2b\") pod \"kas-connection-checker-685895c7c9-qb946\" (UID: \"f50eed36-43f3-4f6d-8aa0-46a148ca8177\") " pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815726    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-serving-cert\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3827824-759c-446a-b653-bed9ccf417c2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.815894 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.816402 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.816402 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815939    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-config\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.816402 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.815958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.816402 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.816020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.816402 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.816102    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlkt\" (UniqueName: \"kubernetes.io/projected/6170d15e-a50a-4912-bfec-932baad8ed5a-kube-api-access-vwlkt\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.816402 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.816133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4k9\" (UniqueName: \"kubernetes.io/projected/3d078cc9-6850-4884-b8d2-b7e5b87f087e-kube-api-access-gz4k9\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.818542 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.818519    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-serving-cert\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.818809 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.818768    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4e32724-e514-41cc-b175-59da7bfd7b8b-serving-cert\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.819061 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.819040    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3827824-759c-446a-b653-bed9ccf417c2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.819137 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.819079    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-stats-auth\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.819381 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.819365    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-default-certificate\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.828317 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.828277    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zncwb\""
Nov 03 15:07:53.828317 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.828278    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tqtrg\""
Nov 03 15:07:53.828317 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.828289    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ncdvq\""
Nov 03 15:07:53.828506 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.828289    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-24jgf\""
Nov 03 15:07:53.855946 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.855920    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 15:07:53.859731 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.859710    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 15:07:53.865362 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.865334    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8db8c20d-d878-475b-97d0-6cd01befd3ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.899244 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.899205    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 15:07:53.899244 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.899219    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 15:07:53.899767 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.899748    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 15:07:53.906267 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.906232    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-trusted-ca\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.906400 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.906304    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kz2vz"]
Nov 03 15:07:53.907175 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.907155    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64c988bcf7-rdfnp"]
Nov 03 15:07:53.907836 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.907814    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp85d\" (UniqueName: \"kubernetes.io/projected/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-kube-api-access-pp85d\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.908038 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.907913    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbgm\" (UniqueName: \"kubernetes.io/projected/8db8c20d-d878-475b-97d0-6cd01befd3ae-kube-api-access-jkbgm\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:53.908038 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.908091    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpnz\" (UniqueName: \"kubernetes.io/projected/f3827824-759c-446a-b653-bed9ccf417c2-kube-api-access-2dpnz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.908309 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.908277    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/5e1df8e2-38c0-4e61-a117-f3b797ec963b-kube-api-access-ncmgr\") pod \"kas-connection-checker-685895c7c9-lxnsf\" (UID: \"5e1df8e2-38c0-4e61-a117-f3b797ec963b\") " pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:07:53.908309 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.908302    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2pq\" (UniqueName: \"kubernetes.io/projected/852144f2-7df8-4279-bf86-65c5351c8945-kube-api-access-2b2pq\") pod \"kas-connection-checker-685895c7c9-gh68l\" (UID: \"852144f2-7df8-4279-bf86-65c5351c8945\") " pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:07:53.908745 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.908721    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bn2b\" (UniqueName: \"kubernetes.io/projected/f50eed36-43f3-4f6d-8aa0-46a148ca8177-kube-api-access-5bn2b\") pod \"kas-connection-checker-685895c7c9-qb946\" (UID: \"f50eed36-43f3-4f6d-8aa0-46a148ca8177\") " pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:07:53.909682 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.909661    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdjh\" (UniqueName: \"kubernetes.io/projected/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-kube-api-access-djdjh\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:53.909946 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.909929    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhdj\" (UniqueName: \"kubernetes.io/projected/a4e32724-e514-41cc-b175-59da7bfd7b8b-kube-api-access-6hhdj\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:53.910176 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.910150    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvvj\" (UniqueName: \"kubernetes.io/projected/1df716cc-1298-4e45-bc66-f8d50e25353f-kube-api-access-phvvj\") pod \"volume-data-source-validator-58c9f5f46d-nn7mt\" (UID: \"1df716cc-1298-4e45-bc66-f8d50e25353f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:07:53.916991 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.916950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.917102 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.917102 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917102 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917102 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplr8\" (UniqueName: \"kubernetes.io/projected/355ce6d3-5601-4133-b805-b378c9c45dd8-kube-api-access-cplr8\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.917170    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.917173    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.917186    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.917196    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.917255    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.417238092 +0000 UTC m=+31.719900300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:07:53.917297 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.917286    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.417277422 +0000 UTC m=+31.719939613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-tmp\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-snapshots\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlkt\" (UniqueName: \"kubernetes.io/projected/6170d15e-a50a-4912-bfec-932baad8ed5a-kube-api-access-vwlkt\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917534    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4k9\" (UniqueName: \"kubernetes.io/projected/3d078cc9-6850-4884-b8d2-b7e5b87f087e-kube-api-access-gz4k9\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917623    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-cert\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917746    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.917780 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce6d3-5601-4133-b805-b378c9c45dd8-config-volume\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d078cc9-6850-4884-b8d2-b7e5b87f087e-serving-cert\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxn8v\" (UniqueName: \"kubernetes.io/projected/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-kube-api-access-mxn8v\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917919    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917924    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.917996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ce6d3-5601-4133-b805-b378c9c45dd8-tmp-dir\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918012    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-service-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6krj\" (UniqueName: \"kubernetes.io/projected/273b804d-46fc-4201-b6d1-1029b44b301d-kube-api-access-r6krj\") pod \"network-check-source-6f78d59f99-qmkmp\" (UID: \"273b804d-46fc-4201-b6d1-1029b44b301d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1612bb40-f413-40d9-89c2-284ddc106ca7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918303    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ce6d3-5601-4133-b805-b378c9c45dd8-tmp-dir\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918315    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-tmp\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918414    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-snapshots\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.918537 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.918003    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.918503    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.41848541 +0000 UTC m=+31.721147617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.918252    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.918546    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.418535955 +0000 UTC m=+31.721198161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918758    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918837    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.918885    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1612bb40-f413-40d9-89c2-284ddc106ca7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.919068    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.919549 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.919400    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-service-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.920007 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.919721    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.920007 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.919989    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.920100 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.920061    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.920324 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.920204    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce6d3-5601-4133-b805-b378c9c45dd8-config-volume\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.920324 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.920219    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:07:53.920324 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:53.920299    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:54.420282745 +0000 UTC m=+31.722944939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:07:53.920324 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.920302    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.921464 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.921437    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.921539 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.921504    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d078cc9-6850-4884-b8d2-b7e5b87f087e-serving-cert\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.922115 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.922093    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-cert\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.922496 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.922478    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.922805 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.922787    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.953821 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.953788    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:07:53.959462 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.959437    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:07:53.974730 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.974703    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:07:53.980359 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.980299    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.980646 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.980537    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.980736 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.980712    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:53.980863 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.980837    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:07:53.981018 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.980901    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:53.981341 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.981321    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplr8\" (UniqueName: \"kubernetes.io/projected/355ce6d3-5601-4133-b805-b378c9c45dd8-kube-api-access-cplr8\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:53.984039 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.983945    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4k9\" (UniqueName: \"kubernetes.io/projected/3d078cc9-6850-4884-b8d2-b7e5b87f087e-kube-api-access-gz4k9\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:53.986259 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.985267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxn8v\" (UniqueName: \"kubernetes.io/projected/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-kube-api-access-mxn8v\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:53.986259 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.985864    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlkt\" (UniqueName: \"kubernetes.io/projected/6170d15e-a50a-4912-bfec-932baad8ed5a-kube-api-access-vwlkt\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:53.988013 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.987954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:07:53.991718 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:53.991691    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6krj\" (UniqueName: \"kubernetes.io/projected/273b804d-46fc-4201-b6d1-1029b44b301d-kube-api-access-r6krj\") pod \"network-check-source-6f78d59f99-qmkmp\" (UID: \"273b804d-46fc-4201-b6d1-1029b44b301d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:07:54.002693 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.002670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:07:54.011874 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.011662    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:07:54.019422 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.019303    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:07:54.052301 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.052060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:07:54.064800 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.064419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:07:54.327225 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.326987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:54.327225 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.327157    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.327133663 +0000 UTC m=+32.629795871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:07:54.327946 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.327269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:54.327946 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.327316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:54.327946 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.327406    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:54.327946 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.327411    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:07:54.327946 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.327446    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.32743687 +0000 UTC m=+32.630099060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:54.327946 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.327466    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.327451629 +0000 UTC m=+32.630113834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:07:54.354207 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.353708    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"]
Nov 03 15:07:54.356829 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.356804    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"]
Nov 03 15:07:54.356949 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.356809    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea757654_0dab_4eb1_bf75_1a38ebf1fb47.slice/crio-c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8 WatchSource:0}: Error finding container c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8: Status 404 returned error can't find the container with id c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8
Nov 03 15:07:54.359327 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.359290    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3827824_759c_446a_b653_bed9ccf417c2.slice/crio-9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3 WatchSource:0}: Error finding container 9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3: Status 404 returned error can't find the container with id 9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3
Nov 03 15:07:54.378726 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.378696    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qb946"]
Nov 03 15:07:54.382290 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.381995    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp" event={"ID":"ea757654-0dab-4eb1-bf75-1a38ebf1fb47","Type":"ContainerStarted","Data":"c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8"}
Nov 03 15:07:54.383555 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.383535    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl" event={"ID":"f3827824-759c-446a-b653-bed9ccf417c2","Type":"ContainerStarted","Data":"9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3"}
Nov 03 15:07:54.383676 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.383654    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50eed36_43f3_4f6d_8aa0_46a148ca8177.slice/crio-3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073 WatchSource:0}: Error finding container 3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073: Status 404 returned error can't find the container with id 3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073
Nov 03 15:07:54.423947 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.421932    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gh68l"]
Nov 03 15:07:54.429155 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.429110    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1df8e2_38c0_4e61_a117_f3b797ec963b.slice/crio-8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9 WatchSource:0}: Error finding container 8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9: Status 404 returned error can't find the container with id 8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9
Nov 03 15:07:54.429551 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.429521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:54.429681 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.429664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:54.429737 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.429703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:54.429737 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429669    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:07:54.429737 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429722    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:07:54.429875 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.429751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:54.429875 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429778    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.42976022 +0000 UTC m=+32.732422425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:07:54.429875 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429810    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.429801108 +0000 UTC m=+32.732463305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:07:54.429875 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429840    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:54.429875 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.429846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429853    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429904    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429915    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429927    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.429917708 +0000 UTC m=+32.732579914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429858    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.429957    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.429942004 +0000 UTC m=+32.732604221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:07:54.430140 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:54.430006    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:55.429963728 +0000 UTC m=+32.732625947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:07:54.430489 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.430470    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lxnsf"]
Nov 03 15:07:54.434353 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.434332    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852144f2_7df8_4279_bf86_65c5351c8945.slice/crio-a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68 WatchSource:0}: Error finding container a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68: Status 404 returned error can't find the container with id a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68
Nov 03 15:07:54.458145 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.456116    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"]
Nov 03 15:07:54.466231 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.466195    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df716cc_1298_4e45_bc66_f8d50e25353f.slice/crio-9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5 WatchSource:0}: Error finding container 9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5: Status 404 returned error can't find the container with id 9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5
Nov 03 15:07:54.472099 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.472076    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-x66cq"]
Nov 03 15:07:54.488228 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.487630    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-w7zh5"]
Nov 03 15:07:54.493133 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.493102    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d078cc9_6850_4884_b8d2_b7e5b87f087e.slice/crio-29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea WatchSource:0}: Error finding container 29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea: Status 404 returned error can't find the container with id 29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea
Nov 03 15:07:54.502717 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.502379    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kz2vz"]
Nov 03 15:07:54.508436 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:54.508369    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"]
Nov 03 15:07:54.510833 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.510800    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6edeeb_d5df_4d05_bd14_f0d3a3d84cff.slice/crio-9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb WatchSource:0}: Error finding container 9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb: Status 404 returned error can't find the container with id 9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb
Nov 03 15:07:54.519237 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:54.519209    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273b804d_46fc_4201_b6d1_1029b44b301d.slice/crio-52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38 WatchSource:0}: Error finding container 52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38: Status 404 returned error can't find the container with id 52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38
Nov 03 15:07:55.160325 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.160293    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:55.160478 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.160294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:55.160556 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.160302    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:55.213809 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.213783    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 15:07:55.213963 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.213857    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z77jb\""
Nov 03 15:07:55.213963 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.213878    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qbvnh\""
Nov 03 15:07:55.213963 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.213964    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-975kv\""
Nov 03 15:07:55.214113 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.213988    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 15:07:55.339081 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.339051    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.339161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.339218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.339247    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.339224768 +0000 UTC m=+34.641886973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.339317    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.339332    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.339369    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.339355476 +0000 UTC m=+34.642017681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:55.339504 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.339388    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.339377565 +0000 UTC m=+34.642039756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:07:55.390321 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.390284    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gh68l" event={"ID":"852144f2-7df8-4279-bf86-65c5351c8945","Type":"ContainerStarted","Data":"9829df38b59ba301167a698c7d23c83c74faac7d5ce98bdd6e9a91e2a58f1bc2"}
Nov 03 15:07:55.390482 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.390347    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gh68l" event={"ID":"852144f2-7df8-4279-bf86-65c5351c8945","Type":"ContainerStarted","Data":"a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68"}
Nov 03 15:07:55.398320 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.398285    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lxnsf" event={"ID":"5e1df8e2-38c0-4e61-a117-f3b797ec963b","Type":"ContainerStarted","Data":"94eb73eb47696b5c29c49a6b55ca0315535a62edbbff1a43e8d53d09683bad06"}
Nov 03 15:07:55.398501 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.398341    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lxnsf" event={"ID":"5e1df8e2-38c0-4e61-a117-f3b797ec963b","Type":"ContainerStarted","Data":"8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9"}
Nov 03 15:07:55.400160 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.400122    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w7zh5" event={"ID":"3d078cc9-6850-4884-b8d2-b7e5b87f087e","Type":"ContainerStarted","Data":"29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea"}
Nov 03 15:07:55.402631 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.402570    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt" event={"ID":"1df716cc-1298-4e45-bc66-f8d50e25353f","Type":"ContainerStarted","Data":"9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5"}
Nov 03 15:07:55.405399 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.405369    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" event={"ID":"a4e32724-e514-41cc-b175-59da7bfd7b8b","Type":"ContainerStarted","Data":"1845d32a340f1cfd65f5a417d39263fad880f8a2c6fbfce09ff294f0f0588c4a"}
Nov 03 15:07:55.408658 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.408627    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qb946" event={"ID":"f50eed36-43f3-4f6d-8aa0-46a148ca8177","Type":"ContainerStarted","Data":"6e232ad5f3ea78eefc3bf3c0c942a395d96c0f24be6bdceb6dbd5f475f4f3a41"}
Nov 03 15:07:55.408803 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.408679    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qb946" event={"ID":"f50eed36-43f3-4f6d-8aa0-46a148ca8177","Type":"ContainerStarted","Data":"3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073"}
Nov 03 15:07:55.412111 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.411990    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kz2vz" event={"ID":"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff","Type":"ContainerStarted","Data":"9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb"}
Nov 03 15:07:55.413453 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.413387    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp" event={"ID":"273b804d-46fc-4201-b6d1-1029b44b301d","Type":"ContainerStarted","Data":"52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38"}
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.440089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.440197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.440229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.440250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.440315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440472    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440520    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.440506545 +0000 UTC m=+34.743168750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440561    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440581    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.440575295 +0000 UTC m=+34.743237486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440615    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440633    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.44062782 +0000 UTC m=+34.743290010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440679    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440686    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440704    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.440698575 +0000 UTC m=+34.743360767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440737    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:55.441694 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440767    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:07:55.442763 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:55.440818    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:07:57.440811509 +0000 UTC m=+34.743473714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:07:55.994593 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:55.994527    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qb946" podStartSLOduration=277.994509562 podStartE2EDuration="4m37.994509562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:55.993927261 +0000 UTC m=+33.296589475" watchObservedRunningTime="2026-11-03 15:07:55.994509562 +0000 UTC m=+33.297171792"
Nov 03 15:07:56.113271 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.113211    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lxnsf" podStartSLOduration=278.11319212 podStartE2EDuration="4m38.11319212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:56.111543314 +0000 UTC m=+33.414205527" watchObservedRunningTime="2026-11-03 15:07:56.11319212 +0000 UTC m=+33.415854339"
Nov 03 15:07:56.147333 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.147281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:07:56.147512 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.147478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:56.148602 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:56.148280    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 15:07:56.148602 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:56.148362    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:08:28.148340338 +0000 UTC m=+65.451002529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : secret "metrics-daemon-secret" not found
Nov 03 15:07:56.154854 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.154818    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:56.229758 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.229151    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gh68l" podStartSLOduration=278.229137754 podStartE2EDuration="4m38.229137754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:56.22750639 +0000 UTC m=+33.530168606" watchObservedRunningTime="2026-11-03 15:07:56.229137754 +0000 UTC m=+33.531799965"
Nov 03 15:07:56.249229 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.248587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:56.268559 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.268439    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:56.379609 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.379349    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:07:56.383013 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.382865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:07:56.834487 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.834322    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8lrk7"]
Nov 03 15:07:56.847309 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:56.847037    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xnsmp"]
Nov 03 15:07:56.847926 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:07:56.847869    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7351fb71_99f2_4358_bd82_895d4ffd76d9.slice/crio-756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6 WatchSource:0}: Error finding container 756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6: Status 404 returned error can't find the container with id 756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6
Nov 03 15:07:57.360822 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.360781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:57.361025 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.360917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:07:57.361025 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.360994    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:07:57.361180 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.361169    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:57.361251 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.361236    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.361215736 +0000 UTC m=+38.663877947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:07:57.361682 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.361664    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.361644327 +0000 UTC m=+38.664306580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:07:57.361779 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.361745    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:07:57.361831 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.361783    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.361770978 +0000 UTC m=+38.664433186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:07:57.430785 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.430670    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xnsmp" event={"ID":"1e3decdb-1a61-4186-9f1d-c41cac5bb6da","Type":"ContainerStarted","Data":"2b1f8ba2f6c806c7ccf4a827acada58627e217ef06fec9a4eb92ba499440a6b9"}
Nov 03 15:07:57.433831 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.433801    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt" event={"ID":"1df716cc-1298-4e45-bc66-f8d50e25353f","Type":"ContainerStarted","Data":"57f59f1cf79aff61922e231783e747e3083caf555a7ce4144c11dd3f4a702c77"}
Nov 03 15:07:57.437204 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.437153    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8lrk7" event={"ID":"7351fb71-99f2-4358-bd82-895d4ffd76d9","Type":"ContainerStarted","Data":"756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6"}
Nov 03 15:07:57.461995 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.461942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:07:57.462149 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.462031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:07:57.462149 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.462070    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:07:57.462149 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.462143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:07:57.462376 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.462242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:07:57.462376 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462361    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:07:57.462475 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462414    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.462401642 +0000 UTC m=+38.765063846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:07:57.462475 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462467    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:07:57.462591 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462502    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.46249131 +0000 UTC m=+38.765153515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:07:57.462591 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462568    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:57.462591 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462580    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:07:57.462738 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462611    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.462600873 +0000 UTC m=+38.765263079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:07:57.462738 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462667    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:07:57.462738 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462678    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:07:57.462738 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462709    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.462699827 +0000 UTC m=+38.765362033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:07:57.462946 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462771    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:07:57.462946 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:07:57.462800    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:01.462791377 +0000 UTC m=+38.765453571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:07:57.551227 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:07:57.551165    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt" podStartSLOduration=222.325989192 podStartE2EDuration="3m44.551144535s" totalImagesPullingTime="2.225155343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:07:57.551029236 +0000 UTC m=+34.853691452" watchObservedRunningTime="2026-11-03 15:07:57.551144535 +0000 UTC m=+34.853806749"
Nov 03 15:08:00.457087 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.457043    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp" event={"ID":"ea757654-0dab-4eb1-bf75-1a38ebf1fb47","Type":"ContainerStarted","Data":"14e0c4446f8f65b570bd3b64e22caf1c7d2e0c22793d19547bd4fd032cf5add0"}
Nov 03 15:08:00.459547 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.459516    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w7zh5" event={"ID":"3d078cc9-6850-4884-b8d2-b7e5b87f087e","Type":"ContainerStarted","Data":"5f1651b6e8eb5d46c4f7e57a1dd44dae1461699c2ab954d28bf14f1a3a7e20ca"}
Nov 03 15:08:00.462863 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.462839    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/0.log"
Nov 03 15:08:00.463001 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.462880    2573 generic.go:356] "Generic (PLEG): container finished" podID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerID="ceffb29d12af75ac7f6d7ffffa5a72ff93328c521664f348639e5f6ae0b6ddb0" exitCode=255
Nov 03 15:08:00.463001 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.462909    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" event={"ID":"a4e32724-e514-41cc-b175-59da7bfd7b8b","Type":"ContainerDied","Data":"ceffb29d12af75ac7f6d7ffffa5a72ff93328c521664f348639e5f6ae0b6ddb0"}
Nov 03 15:08:00.463182 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.463164    2573 scope.go:122] "RemoveContainer" containerID="ceffb29d12af75ac7f6d7ffffa5a72ff93328c521664f348639e5f6ae0b6ddb0"
Nov 03 15:08:00.695088 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.695031    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp" podStartSLOduration=223.005559129 podStartE2EDuration="3m48.695011863s" totalImagesPullingTime="5.689452734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:00.575276466 +0000 UTC m=+37.877938678" watchObservedRunningTime="2026-11-03 15:08:00.695011863 +0000 UTC m=+37.997674077"
Nov 03 15:08:00.817652 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:00.817585    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-w7zh5" podStartSLOduration=222.328731356 podStartE2EDuration="3m47.81756489s" totalImagesPullingTime="5.488833534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:00.696823097 +0000 UTC m=+37.999485294" watchObservedRunningTime="2026-11-03 15:08:00.81756489 +0000 UTC m=+38.120227115"
Nov 03 15:08:01.410346 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.410307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:01.410524 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.410396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:01.410524 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.410432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:01.410677 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.410566    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:01.410677 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.410615    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.410602647 +0000 UTC m=+46.713264851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:01.411043 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.410992    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.41095973 +0000 UTC m=+46.713621936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:08:01.411211 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.411071    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:08:01.411211 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.411105    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.411096379 +0000 UTC m=+46.713758583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:08:01.458846 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.458802    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fnprj"]
Nov 03 15:08:01.473413 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.473381    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/1.log"
Nov 03 15:08:01.474112 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.474080    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/0.log"
Nov 03 15:08:01.474258 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.474127    2573 generic.go:356] "Generic (PLEG): container finished" podID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc" exitCode=255
Nov 03 15:08:01.474609 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.474590    2573 scope.go:122] "RemoveContainer" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc"
Nov 03 15:08:01.474801 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.474782    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-x66cq_openshift-console-operator(a4e32724-e514-41cc-b175-59da7bfd7b8b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b"
Nov 03 15:08:01.479559 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.479527    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kz2vz" event={"ID":"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff","Type":"ContainerStarted","Data":"984c6e14ceced762a68d421e7062286ce2ae190cc09d3f7fc007014236b3f9d4"}
Nov 03 15:08:01.479691 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.479571    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl" event={"ID":"f3827824-759c-446a-b653-bed9ccf417c2","Type":"ContainerStarted","Data":"38d444428fa246d7baaf872350925afbe1d7cbcb20e1aa0832bdee78d8027a21"}
Nov 03 15:08:01.479691 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.479596    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" event={"ID":"a4e32724-e514-41cc-b175-59da7bfd7b8b","Type":"ContainerDied","Data":"4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc"}
Nov 03 15:08:01.479691 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.479625    2573 scope.go:122] "RemoveContainer" containerID="ceffb29d12af75ac7f6d7ffffa5a72ff93328c521664f348639e5f6ae0b6ddb0"
Nov 03 15:08:01.479691 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.479683    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.510920 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.510886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:01.511117 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.510928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:01.511117 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.510993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:01.511117 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.511024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:01.511117 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.511086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511085    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511146    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511159    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511180    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.511161097 +0000 UTC m=+46.813823308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511219    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.511203423 +0000 UTC m=+46.813865633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511283    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511297    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511309    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:08:01.511320 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511315    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.511303686 +0000 UTC m=+46.813965921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:08:01.511705 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511343    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.511331333 +0000 UTC m=+46.813993539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:08:01.511705 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511362    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:08:01.511705 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.511390    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.511383499 +0000 UTC m=+46.814045690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:08:01.522074 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.522039    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fnprj"]
Nov 03 15:08:01.612251 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.612216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqrs\" (UniqueName: \"kubernetes.io/projected/125530fc-c722-456a-8e21-5cae6029db06-kube-api-access-djqrs\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.612423 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.612263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.612423 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.612296    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.612831 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.612790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/125530fc-c722-456a-8e21-5cae6029db06-crio-socket\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.612938 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.612884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/125530fc-c722-456a-8e21-5cae6029db06-data-volume\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.630002 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.629838    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 15:08:01.630002 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.629879    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9729\""
Nov 03 15:08:01.630002 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.629921    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 15:08:01.709714 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.708164    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kz2vz" podStartSLOduration=2.399759156 podStartE2EDuration="8.7081485s" totalImagesPullingTime="6.308389344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:01.587741206 +0000 UTC m=+38.890403420" watchObservedRunningTime="2026-11-03 15:08:01.7081485 +0000 UTC m=+39.010810712"
Nov 03 15:08:01.716326 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqrs\" (UniqueName: \"kubernetes.io/projected/125530fc-c722-456a-8e21-5cae6029db06-kube-api-access-djqrs\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.716501 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.716501 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.716501 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/125530fc-c722-456a-8e21-5cae6029db06-crio-socket\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.716501 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/125530fc-c722-456a-8e21-5cae6029db06-data-volume\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.717059 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.716752    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:01.717059 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.716816    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:02.216796718 +0000 UTC m=+39.519458911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:01.717059 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.716865    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 15:08:01.717059 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:01.716899    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:02.216888089 +0000 UTC m=+39.519550280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : configmap "kube-rbac-proxy" not found
Nov 03 15:08:01.717059 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716860    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/125530fc-c722-456a-8e21-5cae6029db06-crio-socket\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.717059 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.716946    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/125530fc-c722-456a-8e21-5cae6029db06-data-volume\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.787338 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.787297    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqrs\" (UniqueName: \"kubernetes.io/projected/125530fc-c722-456a-8e21-5cae6029db06-kube-api-access-djqrs\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:01.946190 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:01.946121    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl" podStartSLOduration=223.822059255 podStartE2EDuration="3m49.946099704s" totalImagesPullingTime="6.124040449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:01.941330898 +0000 UTC m=+39.243993113" watchObservedRunningTime="2026-11-03 15:08:01.946099704 +0000 UTC m=+39.248761912"
Nov 03 15:08:02.222516 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.222482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:02.222696 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.222523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:02.222696 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:02.222649    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:02.222830 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:02.222697    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 15:08:02.222830 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:02.222734    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.222719342 +0000 UTC m=+40.525381535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:02.222830 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:02.222781    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:03.22276189 +0000 UTC m=+40.525424098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : configmap "kube-rbac-proxy" not found
Nov 03 15:08:02.262617 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.262585    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"]
Nov 03 15:08:02.283957 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.283923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:02.323927 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.323898    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"]
Nov 03 15:08:02.365483 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.365431    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:02.365483 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.365455    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 15:08:02.405125 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.405087    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7mf6v\""
Nov 03 15:08:02.424594 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.424560    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjhm\" (UniqueName: \"kubernetes.io/projected/797abf7a-7927-468d-8a1b-c994b74224db-kube-api-access-8wjhm\") pod \"migrator-66df5d496b-4m6j6\" (UID: \"797abf7a-7927-468d-8a1b-c994b74224db\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:02.479665 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.479596    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/1.log"
Nov 03 15:08:02.480161 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.479994    2573 scope.go:122] "RemoveContainer" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc"
Nov 03 15:08:02.480243 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:02.480217    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-x66cq_openshift-console-operator(a4e32724-e514-41cc-b175-59da7bfd7b8b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b"
Nov 03 15:08:02.481393 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.481359    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp" event={"ID":"273b804d-46fc-4201-b6d1-1029b44b301d","Type":"ContainerStarted","Data":"24ab2e827517f569e26772cc1f8483fc9a425ca90a90f5048ef96b23cb14524c"}
Nov 03 15:08:02.525530 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.525492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjhm\" (UniqueName: \"kubernetes.io/projected/797abf7a-7927-468d-8a1b-c994b74224db-kube-api-access-8wjhm\") pod \"migrator-66df5d496b-4m6j6\" (UID: \"797abf7a-7927-468d-8a1b-c994b74224db\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:02.588150 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.588111    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjhm\" (UniqueName: \"kubernetes.io/projected/797abf7a-7927-468d-8a1b-c994b74224db-kube-api-access-8wjhm\") pod \"migrator-66df5d496b-4m6j6\" (UID: \"797abf7a-7927-468d-8a1b-c994b74224db\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:02.613526 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.613491    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:02.723463 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:02.723381    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp" podStartSLOduration=222.51576957 podStartE2EDuration="3m49.723351624s" totalImagesPullingTime="7.207582054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:02.723102735 +0000 UTC m=+40.025764951" watchObservedRunningTime="2026-11-03 15:08:02.723351624 +0000 UTC m=+40.026013838"
Nov 03 15:08:03.008988 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.008929    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"]
Nov 03 15:08:03.011198 ip-10-0-134-23 kubenswrapper[2573]: W1103 15:08:03.011158    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797abf7a_7927_468d_8a1b_c994b74224db.slice/crio-4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6 WatchSource:0}: Error finding container 4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6: Status 404 returned error can't find the container with id 4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6
Nov 03 15:08:03.233142 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.233105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:03.233142 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.233151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:03.233399 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:03.233264    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:03.233399 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:03.233300    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 15:08:03.233399 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:03.233326    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:05.23330946 +0000 UTC m=+42.535971651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:03.233399 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:03.233365    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:05.233345981 +0000 UTC m=+42.536008188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : configmap "kube-rbac-proxy" not found
Nov 03 15:08:03.486799 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.486715    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8lrk7" event={"ID":"7351fb71-99f2-4358-bd82-895d4ffd76d9","Type":"ContainerStarted","Data":"ee93b4d3adb770cd95e748b89b816e70b54bb0e2d7174f307b1bfe85e36f8269"}
Nov 03 15:08:03.487245 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.486816    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:08:03.488065 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.488013    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6" event={"ID":"797abf7a-7927-468d-8a1b-c994b74224db","Type":"ContainerStarted","Data":"4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6"}
Nov 03 15:08:03.610877 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:03.610815    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8lrk7" podStartSLOduration=33.619010014 podStartE2EDuration="39.610799251s" totalImagesPullingTime="5.991789237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:03.610111552 +0000 UTC m=+40.912773767" watchObservedRunningTime="2026-11-03 15:08:03.610799251 +0000 UTC m=+40.913461522"
Nov 03 15:08:04.003872 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:04.003829    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:04.004079 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:04.003895    2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:04.004352 ip-10-0-134-23 kubenswrapper[2573]: I1103 15:08:04.004329    2573 scope.go:122] "RemoveContainer" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc"
Nov 03 15:08:04.004564 ip-10-0-134-23 kubenswrapper[2573]: E1103 15:08:04.004541    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-x66cq_openshift-console-operator(a4e32724-e514-41cc-b175-59da7bfd7b8b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b"
Nov 03 15:08:04.996259 ip-10-0-134-23 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 15:08:05.014516 ip-10-0-134-23 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 15:08:05.014731 ip-10-0-134-23 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 15:08:05.014991 ip-10-0-134-23 systemd[1]: kubelet.service: Consumed 3.163s CPU time.
Nov 03 15:08:05.026505 ip-10-0-134-23 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 15:08:05.109984 ip-10-0-134-23 kubenswrapper[4955]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 15:08:05.109984 ip-10-0-134-23 kubenswrapper[4955]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 15:08:05.109984 ip-10-0-134-23 kubenswrapper[4955]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 15:08:05.109984 ip-10-0-134-23 kubenswrapper[4955]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 15:08:05.112138 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112044    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.112138 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112139    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.112138 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112143    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112146    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112149    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112152    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112154    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112157    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112159    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112162    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112165    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112167    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112170    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112173    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112175    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112178    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112180    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112183    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112186    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112188    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112190    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112193    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.112234 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112195    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112198    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112200    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112203    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112205    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112212    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112215    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112217    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112220    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112223    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112225    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112228    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112231    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112234    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112237    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112240    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112242    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112245    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112248    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112250    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.112869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112253    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112256    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112259    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112262    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112264    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112267    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112270    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112272    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112275    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112277    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112280    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112283    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112285    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112288    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112292    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112296    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112300    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112306    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.113412 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112310    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112313    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112317    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112319    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112322    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112324    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112327    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112330    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112332    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112335    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112337    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112340    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112342    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112345    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112347    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112350    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112353    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112355    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112358    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112360    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.113849 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112363    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112365    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112367    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112370    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112373    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112375    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112378    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112381    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112383    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112385    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112388    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112391    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112393    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112396    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112398    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112401    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112403    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112406    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112408    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112411    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.114407 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112413    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112416    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112419    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112421    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112424    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.112426    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113220    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113227    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113230    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113241    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113244    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113247    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113251    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113254    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113256    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113259    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113262    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113264    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113267    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113269    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.114890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113272    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113274    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113277    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113279    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113281    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113284    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113287    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113289    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113291    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113294    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113297    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113299    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113302    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113305    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113308    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113310    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113313    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113315    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113320    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.115435 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113323    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113327    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113330    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113338    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113341    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113343    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113346    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113348    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113351    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113353    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113358    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113362    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113364    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113369    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113372    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113374    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113377    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113379    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113382    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113384    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.115897 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113387    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113389    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113392    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113395    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113397    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113400    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113403    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113405    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113407    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113410    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113412    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113415    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113418    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113420    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113423    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113426    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113434    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113437    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113440    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113442    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.116424 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113445    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113447    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113449    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113452    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113454    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113460    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113463    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113465    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113468    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113470    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113473    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113476    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113478    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113481    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113483    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113486    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113488    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113490    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113493    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113496    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.116903 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113498    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113502    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113504    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113507    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113509    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113511    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113514    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113516    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113519    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113522    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113524    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113527    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.113529    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114056    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114063    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114067    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114070    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114075    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114078    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.117443 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114081    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114083    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114086    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114091    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114094    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114097    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114099    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114102    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114104    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114107    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114110    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114112    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114114    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114117    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114120    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114122    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114125    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114127    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114130    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.117894 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114132    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114135    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114139    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114145    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114148    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114152    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114155    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114158    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114161    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114163    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114174    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114179    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114181    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114184    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114186    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114189    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114192    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114196    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114200    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.118382 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114203    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114206    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114209    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114211    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114215    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114217    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114220    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114223    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114225    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114228    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114230    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114233    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114235    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114238    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114240    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114243    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114246    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114249    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114252    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.118860 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114255    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114257    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114259    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114262    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114264    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114269    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114272    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114275    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114278    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114281    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114284    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114287    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114290    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114292    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114295    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114297    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114300    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114302    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114305    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.119330 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114307    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114310    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114312    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114315    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114318    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114320    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114323    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114325    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114328    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114330    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114333    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114335    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114342    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114345    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114347    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114350    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114353    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114356    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114359    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114362    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114365    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.119850 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114367    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114370    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.114373    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114484    4955 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114500    4955 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114511    4955 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114516    4955 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114521    4955 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114524    4955 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114528    4955 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114533    4955 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114536    4955 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114539    4955 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114543    4955 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114546    4955 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114549    4955 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114552    4955 flags.go:64] FLAG: --cgroup-root=""
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114555    4955 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114558    4955 flags.go:64] FLAG: --client-ca-file=""
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114561    4955 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114564    4955 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114568    4955 flags.go:64] FLAG: --cluster-domain=""
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114571    4955 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114574    4955 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 15:08:05.120440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114580    4955 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114584    4955 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114588    4955 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114592    4955 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114595    4955 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114599    4955 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114602    4955 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114605    4955 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114608    4955 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114611    4955 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114615    4955 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114619    4955 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114622    4955 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114625    4955 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114628    4955 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114631    4955 flags.go:64] FLAG: --enable-server="true"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114634    4955 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114643    4955 flags.go:64] FLAG: --event-burst="100"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114647    4955 flags.go:64] FLAG: --event-qps="50"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114650    4955 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114653    4955 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114656    4955 flags.go:64] FLAG: --eviction-hard=""
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114660    4955 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114664    4955 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114667    4955 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 15:08:05.121111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114669    4955 flags.go:64] FLAG: --eviction-soft=""
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114672    4955 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114675    4955 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114678    4955 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114681    4955 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114684    4955 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114687    4955 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114690    4955 flags.go:64] FLAG: --feature-gates=""
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114697    4955 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114700    4955 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114704    4955 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114707    4955 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114710    4955 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114713    4955 flags.go:64] FLAG: --help="false"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114715    4955 flags.go:64] FLAG: --hostname-override="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114719    4955 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114721    4955 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114726    4955 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114729    4955 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114732    4955 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114735    4955 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114738    4955 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114740    4955 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 15:08:05.121757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114743    4955 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114746    4955 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114750    4955 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114753    4955 flags.go:64] FLAG: --kube-reserved=""
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114756    4955 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114759    4955 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114762    4955 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114765    4955 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114768    4955 flags.go:64] FLAG: --lock-file=""
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114771    4955 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114774    4955 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114777    4955 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114782    4955 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114785    4955 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114788    4955 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114791    4955 flags.go:64] FLAG: --logging-format="text"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114794    4955 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114798    4955 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114803    4955 flags.go:64] FLAG: --manifest-url=""
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114806    4955 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114810    4955 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114813    4955 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114817    4955 flags.go:64] FLAG: --max-pods="110"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114820    4955 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114823    4955 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 15:08:05.122324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114826    4955 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114830    4955 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114833    4955 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114836    4955 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114839    4955 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114846    4955 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114849    4955 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114852    4955 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114855    4955 flags.go:64] FLAG: --pod-cidr=""
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114858    4955 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114860    4955 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114863    4955 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114866    4955 flags.go:64] FLAG: --port="10250"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114869    4955 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114872    4955 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02a26c39a727b0b5a"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114875    4955 flags.go:64] FLAG: --qos-reserved=""
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114878    4955 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114881    4955 flags.go:64] FLAG: --register-node="true"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114884    4955 flags.go:64] FLAG: --register-with-taints=""
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114888    4955 flags.go:64] FLAG: --registry-burst="10"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114891    4955 flags.go:64] FLAG: --registry-qps="5"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114894    4955 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114897    4955 flags.go:64] FLAG: --reserved-memory=""
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114902    4955 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114905    4955 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114908    4955 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 15:08:05.122922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114912    4955 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114915    4955 flags.go:64] FLAG: --runonce="false"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114918    4955 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114921    4955 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114924    4955 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114926    4955 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114929    4955 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114933    4955 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114936    4955 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114939    4955 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114943    4955 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114947    4955 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114949    4955 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114952    4955 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114955    4955 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114958    4955 flags.go:64] FLAG: --system-cgroups=""
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114960    4955 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114980    4955 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114983    4955 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114986    4955 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114993    4955 flags.go:64] FLAG: --tls-min-version=""
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114996    4955 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.114999    4955 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115002    4955 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115005    4955 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 15:08:05.123575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115008    4955 flags.go:64] FLAG: --v="2"
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115012    4955 flags.go:64] FLAG: --version="false"
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115017    4955 flags.go:64] FLAG: --vmodule=""
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115021    4955 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115024    4955 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115177    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115181    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115185    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115189    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115193    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115196    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115199    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115202    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115205    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115207    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115210    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115213    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115216    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115219    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115222    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.124193 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115225    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115227    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115230    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115232    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115236    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115238    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115241    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115243    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115246    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115249    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115251    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115254    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115256    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115259    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115261    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115264    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115266    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115269    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115271    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.124959 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115274    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115276    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115280    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115282    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115285    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115287    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115291    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115294    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115298    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115301    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115305    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115307    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115310    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115313    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115316    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115318    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115321    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115324    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115326    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115329    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.125649 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115331    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115334    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115336    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115338    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115341    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115344    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115346    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115348    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115351    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115353    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115356    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115358    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115360    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115363    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115366    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115369    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115372    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115376    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115378    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.126175 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115381    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115383    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115386    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115392    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115394    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115397    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115399    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115402    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115405    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115407    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115410    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115414    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115416    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115419    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115421    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115424    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115426    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115429    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115432    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115435    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.126918 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115437    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115440    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115442    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115445    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115448    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115450    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115453    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115456    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115458    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115461    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115464    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115466    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115469    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.115497    4955 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115534    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115540    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.127559 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115544    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115548    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115551    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115554    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115556    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115559    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115562    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115565    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115569    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115572    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115574    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115577    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115580    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115582    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115585    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115587    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115590    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115593    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115595    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115597    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.128106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115600    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115603    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115605    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115607    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115610    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115612    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115616    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115618    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115621    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115624    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115626    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115631    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115634    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115636    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115639    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115641    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115644    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115646    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115651    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.128631 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115654    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115658    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115660    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115663    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115666    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115668    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115671    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115673    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115676    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115678    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115680    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115683    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115686    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115689    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115691    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115693    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115696    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115698    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115701    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115703    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.129370 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115706    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115708    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115711    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115713    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115717    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115720    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115722    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115725    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115727    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115730    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115732    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115735    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115737    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115741    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115744    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115746    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115749    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115751    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115754    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115757    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.129955 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115759    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115761    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115764    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115766    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115769    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115772    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115774    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115776    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115779    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115781    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115784    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115786    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115789    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115792    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115794    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115797    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115801    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115803    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115805    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115808    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.130723 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115810    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115814    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115816    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115818    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.115821    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.119763    4955 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.119781    4955 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.119801    4955 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.119806    4955 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119865    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119870    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119874    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119877    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119879    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119882    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119884    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119887    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119889    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119892    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.131221 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119895    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119898    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119900    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119903    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119905    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119908    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119912    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119915    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119918    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119920    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119923    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119925    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119928    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119931    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119933    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119936    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119939    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119941    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119943    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.131677 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119946    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119948    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119952    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119956    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119959    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119961    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119964    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119991    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119994    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.119997    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120000    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120002    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120005    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120007    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120010    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120012    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120015    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120017    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.132178 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120019    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120022    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120025    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120028    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120030    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120033    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120036    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120038    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120041    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120043    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120046    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120048    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120051    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120054    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120056    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120059    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120061    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120064    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120066    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120069    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.132683 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120071    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120074    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120077    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120079    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120082    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120084    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120087    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120089    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120091    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120094    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120098    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120101    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120104    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120107    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120109    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120112    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120114    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120117    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120120    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120122    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.133185 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120125    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120128    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120130    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120133    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120136    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120138    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120140    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120143    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120145    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120148    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120150    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120153    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120156    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120158    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120161    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120164    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120166    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120169    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.133675 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120171    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.120201    4955 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120349    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120354    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120357    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120360    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120364    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120367    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120370    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120373    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120376    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120379    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120381    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120384    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120386    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120389    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.134180 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120391    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120394    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120396    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120399    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120401    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120404    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120406    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120408    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120411    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120414    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120416    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120419    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120421    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120425    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120429    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120432    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120435    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120438    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120441    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.134575 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120443    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120446    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120449    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120451    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120454    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120456    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120459    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120462    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120469    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120472    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120475    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120477    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120480    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120483    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120485    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120488    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120490    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120493    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120495    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120497    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.135080 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120500    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120502    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120505    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120507    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120510    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120512    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120514    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120517    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120519    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120522    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120524    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120527    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120529    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120531    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120534    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120538    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120541    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120544    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120547    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.135755 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120549    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120554    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120557    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120559    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120562    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120565    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120569    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120572    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120574    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120577    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120579    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120582    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120584    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120587    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120589    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120592    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120594    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120597    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120600    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.136440 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120602    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120605    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120607    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120610    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120612    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120615    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120617    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120619    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120622    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120624    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120627    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120629    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120632    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120634    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120638    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.120661    4955 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 15:08:05.137173 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120704    4955 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120708    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120711    4955 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120715    4955 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120718    4955 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120720    4955 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120723    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120725    4955 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120728    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120730    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120733    4955 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120735    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120738    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120740    4955 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120743    4955 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120745    4955 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120748    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120750    4955 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120753    4955 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120755    4955 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 15:08:05.137872 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120758    4955 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120762    4955 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120766    4955 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120769    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120771    4955 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120774    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120776    4955 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120779    4955 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120781    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120784    4955 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120786    4955 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120790    4955 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120792    4955 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120796    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120798    4955 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120803    4955 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120806    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120808    4955 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120811    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 15:08:05.138640 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120813    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120816    4955 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120818    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120821    4955 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120824    4955 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120826    4955 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120829    4955 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120831    4955 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120833    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120836    4955 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120838    4955 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120841    4955 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120843    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120846    4955 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120848    4955 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120851    4955 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120853    4955 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120855    4955 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120858    4955 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120860    4955 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 15:08:05.139200 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120863    4955 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120865    4955 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120868    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120872    4955 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120878    4955 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120881    4955 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120883    4955 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120887    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120891    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120893    4955 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120896    4955 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120898    4955 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120901    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120903    4955 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120906    4955 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120908    4955 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120911    4955 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120913    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120916    4955 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 15:08:05.139689 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120919    4955 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120921    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120924    4955 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120926    4955 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120929    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120931    4955 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120934    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120936    4955 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120939    4955 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120942    4955 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120944    4955 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120947    4955 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120949    4955 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120952    4955 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120954    4955 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120957    4955 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120959    4955 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.120963    4955 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121124    4955 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 15:08:05.140201 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121131    4955 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121134    4955 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121147    4955 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121151    4955 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121155    4955 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121159    4955 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121161    4955 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121164    4955 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:05.121168    4955 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.121240    4955 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.122302    4955 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.122360    4955 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.122569    4955 server.go:1014] "Starting client certificate rotation"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.122714    4955 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.122764    4955 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 15:02:23 +0000 UTC" deadline="2028-07-11 03:57:18.755457062 +0000 UTC"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.122792    4955 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14772h49m13.632666787s"
Nov 03 15:08:05.140730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.123030    4955 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 15:08:05.141251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.125373    4955 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 15:08:05.141251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.127507    4955 log.go:25] "Validated CRI v1 runtime API"
Nov 03 15:08:05.141251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.129127    4955 log.go:25] "Validated CRI v1 image API"
Nov 03 15:08:05.141251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.129849    4955 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 15:08:05.141251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.131755    4955 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 baae2c0b-2d9d-45f9-9871-c6a90e3c9eb1:/dev/nvme0n1p3 fb40d821-1043-4a78-975e-5c1f1fdf5d70:/dev/nvme0n1p4]
Nov 03 15:08:05.141394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.131778    4955 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1845d32a340f1cfd65f5a417d39263fad880f8a2c6fbfce09ff294f0f0588c4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1845d32a340f1cfd65f5a417d39263fad880f8a2c6fbfce09ff294f0f0588c4a/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b1f8ba2f6c806c7ccf4a827acada58627e217ef06fec9a4eb92ba499440a6b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b1f8ba2f6c806c7ccf4a827acada58627e217ef06fec9a4eb92ba499440a6b9/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3467489b98251c4ccb2dfce8777edd2995d6697c463ee5bba8794ee7a3ec07ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3467489b98251c4ccb2dfce8777edd2995d6697c463ee5bba8794ee7a3ec07ef/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fe45434e54cac550be0c55af949b7f67d6ffdd65f1a01dd915bc0d35ed2b879/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fe45434e54cac550be0c55af949b7f67d6ffdd65f1a01dd915bc0d35ed2b879/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b30b6a09896114a41acb1a647b64fc71d56733ea3a173911af2879e3b82800e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b30b6a09896114a41acb1a647b64fc71d56733ea3a173911af2879e3b82800e6/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01ff874b-b837-480b-8adb-266dcaac4f07/volumes/kubernetes.io~projected/kube-api-access-7gb27:{mountpoint:/var/lib/kubelet/pods/01ff874b-b837-480b-8adb-266dcaac4f07/volumes/kubernetes.io~projected/kube-api-access-7gb27 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/125530fc-c722-456a-8e21-5cae6029db06/volumes/kubernetes.io~projected/kube-api-access-djqrs:{mountpoint:/var/lib/kubelet/pods/125530fc-c722-456a-8e21-5cae6029db06/volumes/kubernetes.io~projected/kube-api-access-djqrs major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1df716cc-1298-4e45-bc66-f8d50e25353f/volumes/kubernetes.io~projected/kube-api-access-phvvj:{mountpoint:/var/lib/kubelet/pods/1df716cc-1298-4e45-bc66-f8d50e25353f/volumes/kubernetes.io~projected/kube-api-access-phvvj major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e3decdb-1a61-4186-9f1d-c41cac5bb6da/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1e3decdb-1a61-4186-9f1d-c41cac5bb6da/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/273b804d-46fc-4201-b6d1-1029b44b301d/volumes/kubernetes.io~projected/kube-api-access-r6krj:{mountpoint:/var/lib/kubelet/pods/273b804d-46fc-4201-b6d1-1029b44b301d/volumes/kubernetes.io~projected/kube-api-access-r6krj major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2afcc4bb-db5b-431a-bdb9-580af2eccd18/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2afcc4bb-db5b-431a-bdb9-580af2eccd18/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ede9faa-be51-4210-9fa6-1dc7ddbca38b/volumes/kubernetes.io~projected/kube-api-access-blxt7:{mountpoint:/var/lib/kubelet/pods/2ede9faa-be51-4210-9fa6-1dc7ddbca38b/volumes/kubernetes.io~projected/kube-api-access-blxt7 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/355ce6d3-5601-4133-b805-b378c9c45dd8/volumes/kubernetes.io~projected/kube-api-access-cplr8:{mountpoint:/var/lib/kubelet/pods/355ce6d3-5601-4133-b805-b378c9c45dd8/volumes/kubernetes.io~projected/kube-api-access-cplr8 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d078cc9-6850-4884-b8d2-b7e5b87f087e/volumes/kubernetes.io~projected/kube-api-access-gz4k9:{mountpoint:/var/lib/kubelet/pods/3d078cc9-6850-4884-b8d2-b7e5b87f087e/volumes/kubernetes.io~projected/kube-api-access-gz4k9 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d078cc9-6850-4884-b8d2-b7e5b87f087e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3d078cc9-6850-4884-b8d2-b7e5b87f087e/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55ee9f5d-10b0-4ae3-946b-5295148f2429/volumes/kubernetes.io~projected/kube-api-access-zj98j:{mountpoint:/var/lib/kubelet/pods/55ee9f5d-10b0-4ae3-946b-5295148f2429/volumes/kubernetes.io~projected/kube-api-access-zj98j major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58ea0b3a-4929-431a-8cfe-410028747797/volumes/kubernetes.io~projected/kube-api-access-tdwzx:{mountpoint:/var/lib/kubelet/pods/58ea0b3a-4929-431a-8cfe-410028747797/volumes/kubernetes.io~projected/kube-api-access-tdwzx major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e1df8e2-38c0-4e61-a117-f3b797ec963b/volumes/kubernetes.io~projected/kube-api-access-ncmgr:{mountpoint:/var/lib/kubelet/pods/5e1df8e2-38c0-4e61-a117-f3b797ec963b/volumes/kubernetes.io~projected/kube-api-access-ncmgr major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6170d15e-a50a-4912-bfec-932baad8ed5a/volumes/kubernetes.io~projected/kube-api-access-vwlkt:{mountpoint:/var/lib/kubelet/pods/6170d15e-a50a-4912-bfec-932baad8ed5a/volumes/kubernetes.io~projected/kube-api-access-vwlkt major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7351fb71-99f2-4358-bd82-895d4ffd76d9/volumes/kubernetes.io~projected/kube-api-access-dzkcj:{mountpoint:/var/lib/kubelet/pods/7351fb71-99f2-4358-bd82-895d4ffd76d9/volumes/kubernetes.io~projected/kube-api-access-dzkcj major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/797abf7a-7927-468d-8a1b-c994b74224db/volumes/kubernetes.io~projected/kube-api-access-8wjhm:{mountpoint:/var/lib/kubelet/pods/797abf7a-7927-468d-8a1b-c994b74224db/volumes/kubernetes.io~projected/kube-api-access-8wjhm major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/852144f2-7df8-4279-bf86-65c5351c8945/volumes/kubernetes.io~projected/kube-api-access-2b2pq:{mountpoint:/var/lib/kubelet/pods/852144f2-7df8-4279-bf86-65c5351c8945/volumes/kubernetes.io~projected/kube-api-access-2b2pq major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8db8c20d-d878-475b-97d0-6cd01befd3ae/volumes/kubernetes.io~projected/kube-api-access-jkbgm:{mountpoint:/var/lib/kubelet/pods/8db8c20d-d878-475b-97d0-6cd01befd3ae/volumes/kubernetes.io~projected/kube-api-access-jkbgm major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~projected/kube-api-access-cdvbs:{mountpoint:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~projected/kube-api-access-cdvbs major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/volumes/kubernetes.io~projected/kube-api-access-mxn8v:{mountpoint:/var/lib/kubelet/pods/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/volumes/kubernetes.io~projected/kube-api-access-mxn8v major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~projected/kube-api-access-djdjh:{mountpoint:/var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~projected/kube-api-access-djdjh major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4e32724-e514-41cc-b175-59da7bfd7b8b/volumes/kubernetes.io~projected/kube-api-access-6hhdj:{mountpoint:/var/lib/kubelet/pods/a4e32724-e514-41cc-b175-59da7bfd7b8b/volumes/kubernetes.io~projected/kube-api-access-6hhdj major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4e32724-e514-41cc-b175-59da7bfd7b8b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a4e32724-e514-41cc-b175-59da7bfd7b8b/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b14d1e13-22f0-4ab1-b82c-e1694965e8d8/volumes/kubernetes.io~projected/kube-api-access-zq6zd:{mountpoint:/var/lib/kubelet/pods/b14d1e13-22f0-4ab1-b82c-e1694965e8d8/volumes/kubernetes.io~projected/kube-api-access-zq6zd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc79121f-b8ff-433d-962e-e6c5d02a65f8/volumes/kubernetes.io~projected/kube-api-access-gtchn:{mountpoint:/var/lib/kubelet/pods/bc79121f-b8ff-433d-962e-e6c5d02a65f8/volumes/kubernetes.io~projected/kube-api-access-gtchn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6171645-b2a3-4bf8-8258-a5aef22950e7/volumes/kubernetes.io~projected/kube-api-access-jqx28:{mountpoint:/var/lib/kubelet/pods/c6171645-b2a3-4bf8-8258-a5aef22950e7/volumes/kubernetes.io~projected/kube-api-access-jqx28 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volumes/kubernetes.io~projected/kube-api-access-wg7dn:{mountpoint:/var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volumes/kubernetes.io~projected/kube-api-access-wg7dn major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea757654-0dab-4eb1-bf75-1a38ebf1fb47/volumes/kubernetes.io~projected/kube-api-access-pp85d:{mountpoint:/var/lib/kubelet/pods/ea757654-0dab-4eb1-bf75-1a38ebf1fb47/volumes/kubernetes.io~projected/kube-api-access-pp85d major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea757654-0dab-4eb1-bf75-1a38ebf1fb47/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ea757654-0dab-4eb1-bf75-1a38ebf1fb47/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~projected/kube-api-access-8wlcf:{mountpoint:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~projected/kube-api-access-8wlcf major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3827824-759c-446a-b653-bed9ccf417c2/volumes/kubernetes.io~projected/kube-api-access-2dpnz:{mountpoint:/var/lib/kubelet/pods/f3827824-759c-446a-b653-bed9ccf417c2/volumes/kubernetes.io~projected/kube-api-access-2dpnz major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3827824-759c-446a-b653-bed9ccf417c2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f3827824-759c-446a-b653-bed9ccf417c2/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f50eed36-43f3-4f6d-8aa0-46a148ca8177/volumes/kubernetes.io~projected/kube-api-access-5bn2b:{mountpoint:/var/lib/kubelet/pods/f50eed36-43f3-4f6d-8aa0-46a148ca8177/volumes/kubernetes.io~projected/kube-api-access-5bn2b major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~projected/kube-api-access-69jrl:{mountpoint:/var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~projected/kube-api-access-69jrl major:0 minor:54 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cadcf19d0f4b6f83820bda6b95944b751252023a8dbdf22ed6c78a396250d2ed/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8a2aff9c3a1cdda27fb8bb5bb71f6fa6f182cc6191a2e267a721f5aa695ffe51/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/36d308c42cc7c47b46e2b13c66083efbb423c500a2bc1872674a6bc72ddddb7d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/5f1d57517b5441c02ac2c46d86b03edc5bd937a6a917c5f95f4cb93b69430793/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7597cc96c092ecc676e4bb129da46e3472596a5f808502404e3df10ba32904a3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/48a90c34e7ee82b6fc8660a34515950b3c3a220d4317d3c451a577ad4e761804/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/56542ca60694b66dfc28d517bd2b73c497a13e4380426b6d64541bca292360ee/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5177aff07d66446011b2a071ff0add66fbac86e7bceb8b4538b4a284fabdbf64/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/17639f305b8cbcbb97151521a1874a82faa675207452c4afba61b401a2eafbc8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/131f04510582b2cf8d01ca53b7d69f4f33a6b6e041ae44a21d68ec5b0d13c838/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b3814b4c4c71e1a0f67d79dfdc26c71cd7ee2e73e9c5a25a6aafcb7809408f3a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/f167aae9a1ab1244f70272ba599cef00f68edb9f08aef27f17cab8e4b8fd3136/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ad52402b0956d0a2901d2b8a20c93e1823c1fb452adda8cb3170b81262075087/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/15d1bc3bc0aee0ca14e4477b6a00e0074a1336ebee992a9df463b9930e8c47f0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/817be8521c3b690040b9482be4bc57a1de5783d0fd7b31826e9535cc35b32263/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/03f01ae2d6cecc55da38e3cf225fb77fb6f0f6d43d708b445e84f92cd3712270/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/48ef5ced9ca72c26c868e756edf500658760d34d307adf92bb821d2955776441/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/dfab90b5dee469be144d3a39d9c541accd4515e12882abc6e5c680d86a2bac91/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/482dc371a27acd1a8c5ed241c247f18f4e8209f55f06e6b8bf32cf3c98809570/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d8fc23fbcffb80b25d5671b68b50748362a22e0d68196c30309d1420624a6895/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e82cd5d38ca0105d8b81628ed0786b0d2a55b599997cd6600f837858e65584ef/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/bedd55f6728223cda7e35082bfd7bad736acec9d8036998d09cbb08ca740537e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b2be61777b0d7c13aa6df3b06c96c6b48e76d8a7b9669cf3e17fe9a65ace6d23/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/869116818fa0c05b76fd28882c3c4f3396602639cbfef328301d92e9fbb89538/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f98dca87ddd6d8076c40e68cc2e459a417143e55f3b14c7fa354c30cebb097da/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/535edd6eaba0290b5275da0f904da458acae9a24a128a3a9c6b7fb4440e2dec0/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/a32659bc99d461068e9cc5f6c624027826d0bd3624c54f2ea7be1d04d7ff20d8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/56e23e18e4f7dc696dd2e21bdcfb6d920f01c2c72543ef55e12209ea538e0c14/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d46b42cb64562abb3443daaa2e4f7be1be5ce435c696cffc8f381826bde88fde/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5b79a1c62ab4cb3825984ea39cebef9467663d82053b364ab5ada225b8ed3940/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/62caeeff50bf839a1aa16f38bca35290e41b9b48dece7e6f0174fc1dc6a2cd5b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/3cc394f8f25feab980f60a2a93acb1591824764058683137e5416ed5336f4e40/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/93bd09af154e6e27056c697ddc7e954af50fe552055391da8baa32c7804eb76c/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7de6cb9ceac2e55142cb310577a0a41d2508928cef5b6adb342604242bddfb4d/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/dd41f52192e24f937291b6b061154fb253c232dcba86438fbb2ca90a8dad1e65/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/c3f84cc7d9976e4c80a01f30abc48665d664edcb9aa8b9182d840df6e61acc16/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/39233d4368abf06c1d941b69cb49b32ad6e8f0571cd7aab527c2f611741e9454/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/ca474fa2385f71fe125b1eed86835a014a81bd25318c076d8e5c9c313e0349e1/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/bbe48394dbe83036568e4bce7a8cbd08ed83b9516ce2363caf9c9a76c9421136/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b521d14b13c71b558c673b1172b5232d79c05019a8a7d5eb5172637fa19ef2d2/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/63fb18555f6e42cb06f546ab645b7b08f8a939b0535a291c4530ba2ea5fe686c/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/e73569e788f0f054711364cd4e3a7a92a285643d55193e00721ea21a27eaaad5/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/5022889ad6c13a8d91e65dce23d279c56da07b4cfd45eb2e335a366670e00709/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/4f3621c2d3cd838a15e19bc842bb4a1ddae3c6fc405d0f7c6a7b3026d83abb34/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/135fc72076f0de43ae7407eac8092d152bfb376f9822b4693e487fb9d3f4371d/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/9ccb94f18305940383d7c83e092ea92f16a92f46b8a5786ba56533688c915e97/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/27a0fb245c5eec217f4fcc1f3d55afab68471fef81f846fb9879cec04faabb50/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/99dff9cd29cb8df6eec7180dc7cab55835fbb7f3ae91f5e528af65d94ac424a0/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/726f57b56bacb2bb64c46fde1d22fe0685c578214f59a3c126c6e4b03bfe63e0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/41bb5b4b80f8f7b043e9c71a28674224a708d58a0cedf77b45933a7a56519c01/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1e6ba19e55af3e652791da7b83d46fb3170e224c214523fbad1201a5dc24f6d3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f7c7c5075cb867e60cd2a32de7f89aaa2d3325b6d862bfb80f5a1bf25dc0d697/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/eb3f8c184850c647926fd565c6adafadf97a0dba6fb5b0772f2bda86ff4477df/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/b7f1b446ecd9b1d2821d982f09cde7ab7a9d2da54a8e215a02afecefbfd223c4/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/32ab6f56d9aadbc48286864a62a8f064cf4d8cef96f83a05bfb36bfb47a6869d/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.139216    4955 manager.go:217] Machine: {Timestamp:2026-11-03 15:08:05.138717607 +0000 UTC m=+0.063535510 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099141 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d41e76c2d2310f138a148db54fe0 SystemUUID:ec20d41e-76c2-d231-0f13-8a148db54fe0 BootID:ace703fb-c1b2-4181-8800-a5140debe76f Filesystems:[{Device:/var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c6171645-b2a3-4bf8-8258-a5aef22950e7/volumes/kubernetes.io~projected/kube-api-access-jqx28 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f3827824-759c-446a-b653-bed9ccf417c2/volumes/kubernetes.io~projected/kube-api-access-2dpnz DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b1f8ba2f6c806c7ccf4a827acada58627e217ef06fec9a4eb92ba499440a6b9/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bc79121f-b8ff-433d-962e-e6c5d02a65f8/volumes/kubernetes.io~projected/kube-api-access-gtchn DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3827824-759c-446a-b653-bed9ccf417c2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9aaf1aafc9a9ffeaaa10d50e61a855068959ad45f32ef8438ef4bb7b00393efb/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/273b804d-46fc-4201-b6d1-1029b44b301d/volumes/kubernetes.io~projected/kube-api-access-r6krj DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52792839120bf97db1f8abac2c46ab188f09ebf092cbfde9c3e65bbc16878e38/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7351fb71-99f2-4358-bd82-895d4ffd76d9/volumes/kubernetes.io~projected/kube-api-access-dzkcj DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ede9faa-be51-4210-9fa6-1dc7ddbca38b/volumes/kubernetes.io~projected/kube-api-access-blxt7 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55ee9f5d-10b0-4ae3-946b-5295148f2429/volumes/kubernetes.io~projected/kube-api-access-zj98j DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58ea0b3a-4929-431a-8cfe-410028747797/volumes/kubernetes.io~projected/kube-api-access-tdwzx DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4d8a1fb5dedb2bd920b9befbd416a4f20eeb5a5a823e2ff548e385bbba8e45f/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3747f579da14ef91570964837274cfa5562898485a65e2d77fd451e55cb5e073/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eae286edb7e5ceffad44348168bcef8fe1a0b743779263fcdc80391a0166db9/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e3decdb-1a61-4186-9f1d-c41cac5bb6da/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f94b8f15-cb42-44c9-bbdc-6c609dec30db/volumes/kubernetes.io~projected/kube-api-access-69jrl DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4e32724-e514-41cc-b175-59da7bfd7b8b/volumes/kubernetes.io~projected/kube-api-access-6hhdj DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/volumes/kubernetes.io~projected/kube-api-access-mxn8v DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e9b47e16bf09d2ed28f04c251a5f2fddccd96bfda2a820b2b0c11c1fd0ad36c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f0448e63128ad7a3a24c86945f03808fe1cf360e999ce74b833a1110b61b290/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/125530fc-c722-456a-8e21-5cae6029db06/volumes/kubernetes.io~projected/kube-api-access-djqrs DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cc77a4309b33cea8f52d326422caf9ed92ed0392dcecea5d24eff365ff97be6/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fe45434e54cac550be0c55af949b7f67d6ffdd65f1a01dd915bc0d35ed2b879/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea757654-0dab-4eb1-bf75-1a38ebf1fb47/volumes/kubernetes.io~projected/kube-api-access-pp85d DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d078cc9-6850-4884-b8d2-b7e5b87f087e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/989d17b83a7c005b8d395db03719112b4ff19932d8deaf47c295ecde69bb5974/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea757654-0dab-4eb1-bf75-1a38ebf1fb47/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d0d206593aeb4041d3c2f4619a25210dda4e3d2b6385cc0d22e946e1b317ed3/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2afcc4bb-db5b-431a-bdb9-580af2eccd18/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/355ce6d3-5601-4133-b805-b378c9c45dd8/volumes/kubernetes.io~projected/kube-api-access-cplr8 DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/797abf7a-7927-468d-8a1b-c994b74224db/volumes/kubernetes.io~projected/kube-api-access-8wjhm DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1df716cc-1298-4e45-bc66-f8d50e25353f/volumes/kubernetes.io~projected/kube-api-access-phvvj DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~projected/kube-api-access-cdvbs DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df2075cb93c0501579a456e58ad0ee1874157ba52a9f84c96a8745feeff56737/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4e32724-e514-41cc-b175-59da7bfd7b8b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a947e5c33c4d6e2b156913c576519e66557313bb63b762a6bfe94e2bc15f1a68/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~projected/kube-api-access-djdjh DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d9d6682e019d83204cbff0caaa66f0a2c080019c3c31eddba22bd3f3ec593e5/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bf4fe65379a5cfc974e23fd92d78487fbabcc71a6a229023a93db58e61641f6/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2bc49fe8006842d8ee4b94148e14440125884db90a010b7b7562c20187d210e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8db8c20d-d878-475b-97d0-6cd01befd3ae/volumes/kubernetes.io~projected/kube-api-access-jkbgm DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6170d15e-a50a-4912-bfec-932baad8ed5a/volumes/kubernetes.io~projected/kube-api-access-vwlkt DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3467489b98251c4ccb2dfce8777edd2995d6697c463ee5bba8794ee7a3ec07ef/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1845d32a340f1cfd65f5a417d39263fad880f8a2c6fbfce09ff294f0f0588c4a/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d116bb71-10c3-40c9-ac2d-50a79bc72946/volumes/kubernetes.io~projected/kube-api-access-wg7dn DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c01acdb853f0bd43eb9cbf90d999185598447921c83406a9035a0c7ec13129a8/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01ff874b-b837-480b-8adb-266dcaac4f07/volumes/kubernetes.io~projected/kube-api-access-7gb27 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b30b6a09896114a41acb1a647b64fc71d56733ea3a173911af2879e3b82800e6/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29588fd0589b7de53d2ec4bddcecdafd72245aaf829e5e8683dbf7a17f3a39ea/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b14d1e13-22f0-4ab1-b82c-e1694965e8d8/volumes/kubernetes.io~projected/kube-api-access-zq6zd DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e1df8e2-38c0-4e61-a117-f3b797ec963b/volumes/kubernetes.io~projected/kube-api-access-ncmgr DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f50eed36-43f3-4f6d-8aa0-46a148ca8177/volumes/kubernetes.io~projected/kube-api-access-5bn2b DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes/kubernetes.io~projected/kube-api-access-8wlcf DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/756e7f4f974a06aea5648ad8dd10c32c5445afccdbee416782eb02cc6e0650a6/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/852144f2-7df8-4279-bf86-65c5351c8945/volumes/kubernetes.io~projected/kube-api-access-2b2pq DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d078cc9-6850-4884-b8d2-b7e5b87f087e/volumes/kubernetes.io~projected/kube-api-access-gz4k9 DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d77d0d5-bd4b-423f-b30a-994ad3a5e058/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1845d32a340f1cf MacAddress:8e:5b:99:46:b2:26 Speed:10000 Mtu:8901} {Name:29588fd0589b7de MacAddress:4e:34:0b:41:d2:43 Speed:10000 Mtu:8901} {Name:2b1f8ba2f6c806c MacAddress:a2:93:44:7f:77:f4 Speed:10000 Mtu:8901} {Name:3747f579da14ef9 MacAddress:2e:c2:77:db:3f:8b Speed:10000 Mtu:8901} {Name:4bf4fe65379a5cf MacAddress:9a:5e:51:40:77:67 Speed:10000 Mtu:8901} {Name:52792839120bf97 MacAddress:76:f8:08:67:fd:59 Speed:10000 Mtu:8901} {Name:756e7f4f974a06a MacAddress:7e:3f:e2:2d:af:14 Speed:10000 Mtu:8901} {Name:8eae286edb7e5ce MacAddress:9e:8f:3c:40:fc:0c Speed:10000 Mtu:8901} {Name:9aaf1aafc9a9ffe MacAddress:da:33:34:36:a5:2f Speed:10000 Mtu:8901} {Name:9d0d206593aeb40 MacAddress:d6:03:4b:de:8f:4b Speed:10000 Mtu:8901} {Name:9d9d6682e019d83 MacAddress:76:27:6e:90:21:f5 Speed:10000 Mtu:8901} {Name:a947e5c33c4d6e2 MacAddress:66:fd:b3:89:d4:d9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:6d:3c:c8:36:3d Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:cc:a3:db:af:54 Speed:0 Mtu:8901} {Name:c01acdb853f0bd4 MacAddress:f2:37:60:64:1a:2f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:6d:3c:c8:36:3d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f2:8f:a0:8c:d9:71 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:f7:19:0b:42:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.139570    4955 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.139617    4955 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.139917    4955 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.139938    4955 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-23.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140102    4955 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140111    4955 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140126    4955 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140139    4955 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140237    4955 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140413    4955 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140518    4955 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140552    4955 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140561    4955 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140572    4955 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140587    4955 kubelet.go:405] "Adding apiserver pod source"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.140600    4955 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.141361    4955 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 15:08:05.141599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.141374    4955 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 15:08:05.142508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.142118    4955 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 15:08:05.155919 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.155879    4955 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 15:08:05.155919 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.155903    4955 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 15:08:05.156169 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156066    4955 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 15:08:05.156333 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156316    4955 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 15:08:05.156405 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156357    4955 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 15:08:05.156499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156487    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156506    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156516    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156526    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156536    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156546    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156555    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 15:08:05.156564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156564    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 15:08:05.156888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156576    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 15:08:05.156888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156587    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 15:08:05.156888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156601    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 15:08:05.156888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156616    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 15:08:05.156888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156653    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 15:08:05.156888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.156663    4955 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 15:08:05.160373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.160354    4955 server.go:1263] "Started kubelet"
Nov 03 15:08:05.160483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.160432    4955 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 15:08:05.160997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.160863    4955 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 15:08:05.160997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.160957    4955 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 15:08:05.161290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.161273    4955 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 15:08:05.161445 ip-10-0-134-23 systemd[1]: Started Kubernetes Kubelet.
Nov 03 15:08:05.162361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.162001    4955 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164387    4955 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164403    4955 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164471    4955 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 15:02:24 +0000 UTC" deadline="2028-04-16 23:35:31.286326658 +0000 UTC"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164509    4955 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12728h27m26.121820252s"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164573    4955 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164586    4955 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.164589    4955 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found"
Nov 03 15:08:05.164892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.164633    4955 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 15:08:05.165879 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.165859    4955 factory.go:55] Registering systemd factory
Nov 03 15:08:05.165879 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.165881    4955 factory.go:223] Registration of the systemd container factory successfully
Nov 03 15:08:05.167437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.166494    4955 factory.go:153] Registering CRI-O factory
Nov 03 15:08:05.167437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.166520    4955 factory.go:223] Registration of the crio container factory successfully
Nov 03 15:08:05.167437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.166588    4955 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 15:08:05.167437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.166608    4955 factory.go:103] Registering Raw factory
Nov 03 15:08:05.167437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.166628    4955 manager.go:1199] Started watching for new ooms in manager
Nov 03 15:08:05.167437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.167025    4955 manager.go:319] Starting recovery of all containers
Nov 03 15:08:05.169379 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169336    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 15:08:05.169453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169386    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f94b8f15-cb42-44c9-bbdc-6c609dec30db" volumeName="kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-tuned" seLinuxMountContext=""
Nov 03 15:08:05.169453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169410    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d078cc9-6850-4884-b8d2-b7e5b87f087e" volumeName="kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 15:08:05.169453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169425    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55ee9f5d-10b0-4ae3-946b-5295148f2429" volumeName="kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-cni-binary-copy" seLinuxMountContext=""
Nov 03 15:08:05.169453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169440    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6170d15e-a50a-4912-bfec-932baad8ed5a" volumeName="kubernetes.io/projected/6170d15e-a50a-4912-bfec-932baad8ed5a-kube-api-access-vwlkt" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169459    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169472    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f94b8f15-cb42-44c9-bbdc-6c609dec30db" volumeName="kubernetes.io/projected/f94b8f15-cb42-44c9-bbdc-6c609dec30db-kube-api-access-69jrl" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169494    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="273b804d-46fc-4201-b6d1-1029b44b301d" volumeName="kubernetes.io/projected/273b804d-46fc-4201-b6d1-1029b44b301d-kube-api-access-r6krj" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169513    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d078cc9-6850-4884-b8d2-b7e5b87f087e" volumeName="kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-service-ca-bundle" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169533    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14d1e13-22f0-4ab1-b82c-e1694965e8d8" volumeName="kubernetes.io/projected/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-kube-api-access-zq6zd" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169547    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea757654-0dab-4eb1-bf75-1a38ebf1fb47" volumeName="kubernetes.io/projected/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-kube-api-access-pp85d" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169562    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169580    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc79121f-b8ff-433d-962e-e6c5d02a65f8" volumeName="kubernetes.io/configmap/bc79121f-b8ff-433d-962e-e6c5d02a65f8-serviceca" seLinuxMountContext=""
Nov 03 15:08:05.169616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169595    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db8c20d-d878-475b-97d0-6cd01befd3ae" volumeName="kubernetes.io/configmap/8db8c20d-d878-475b-97d0-6cd01befd3ae-telemetry-config" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169636    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4e32724-e514-41cc-b175-59da7bfd7b8b" volumeName="kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-config" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169649    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169754    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ff874b-b837-480b-8adb-266dcaac4f07" volumeName="kubernetes.io/projected/01ff874b-b837-480b-8adb-266dcaac4f07-kube-api-access-7gb27" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169767    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2afcc4bb-db5b-431a-bdb9-580af2eccd18" volumeName="kubernetes.io/configmap/2afcc4bb-db5b-431a-bdb9-580af2eccd18-konnectivity-ca" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169785    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d078cc9-6850-4884-b8d2-b7e5b87f087e" volumeName="kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-tmp" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169798    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d116bb71-10c3-40c9-ac2d-50a79bc72946" volumeName="kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-env-overrides" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169810    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d078cc9-6850-4884-b8d2-b7e5b87f087e" volumeName="kubernetes.io/projected/3d078cc9-6850-4884-b8d2-b7e5b87f087e-kube-api-access-gz4k9" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169825    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d078cc9-6850-4884-b8d2-b7e5b87f087e" volumeName="kubernetes.io/secret/3d078cc9-6850-4884-b8d2-b7e5b87f087e-serving-cert" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169838    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169854    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14d1e13-22f0-4ab1-b82c-e1694965e8d8" volumeName="kubernetes.io/configmap/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-iptables-alerter-script" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169866    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d116bb71-10c3-40c9-ac2d-50a79bc72946" volumeName="kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169879    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3827824-759c-446a-b653-bed9ccf417c2" volumeName="kubernetes.io/configmap/f3827824-759c-446a-b653-bed9ccf417c2-config" seLinuxMountContext=""
Nov 03 15:08:05.169897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169894    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3decdb-1a61-4186-9f1d-c41cac5bb6da" volumeName="kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169907    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ede9faa-be51-4210-9fa6-1dc7ddbca38b" volumeName="kubernetes.io/projected/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-kube-api-access-blxt7" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169930    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ea0b3a-4929-431a-8cfe-410028747797" volumeName="kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.169944    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170091    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4e32724-e514-41cc-b175-59da7bfd7b8b" volumeName="kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-trusted-ca" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170107    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170126    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f50eed36-43f3-4f6d-8aa0-46a148ca8177" volumeName="kubernetes.io/projected/f50eed36-43f3-4f6d-8aa0-46a148ca8177-kube-api-access-5bn2b" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170140    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6171645-b2a3-4bf8-8258-a5aef22950e7" volumeName="kubernetes.io/projected/c6171645-b2a3-4bf8-8258-a5aef22950e7-kube-api-access-jqx28" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170153    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355ce6d3-5601-4133-b805-b378c9c45dd8" volumeName="kubernetes.io/configmap/355ce6d3-5601-4133-b805-b378c9c45dd8-config-volume" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170181    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355ce6d3-5601-4133-b805-b378c9c45dd8" volumeName="kubernetes.io/empty-dir/355ce6d3-5601-4133-b805-b378c9c45dd8-tmp-dir" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170196    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55ee9f5d-10b0-4ae3-946b-5295148f2429" volumeName="kubernetes.io/projected/55ee9f5d-10b0-4ae3-946b-5295148f2429-kube-api-access-zj98j" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170217    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4e32724-e514-41cc-b175-59da7bfd7b8b" volumeName="kubernetes.io/secret/a4e32724-e514-41cc-b175-59da7bfd7b8b-serving-cert" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170231    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d116bb71-10c3-40c9-ac2d-50a79bc72946" volumeName="kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-config" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170244    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea757654-0dab-4eb1-bf75-1a38ebf1fb47" volumeName="kubernetes.io/configmap/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-config" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170261    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170275    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1612bb40-f413-40d9-89c2-284ddc106ca7" volumeName="kubernetes.io/configmap/1612bb40-f413-40d9-89c2-284ddc106ca7-nginx-conf" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170292    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff" volumeName="kubernetes.io/projected/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-kube-api-access-mxn8v" seLinuxMountContext=""
Nov 03 15:08:05.170337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170309    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff" volumeName="kubernetes.io/secret/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-cert" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170326    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d77d0d5-bd4b-423f-b30a-994ad3a5e058" volumeName="kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-default-certificate" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170375    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170394    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f94b8f15-cb42-44c9-bbdc-6c609dec30db" volumeName="kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-tmp" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170418    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc79121f-b8ff-433d-962e-e6c5d02a65f8" volumeName="kubernetes.io/projected/bc79121f-b8ff-433d-962e-e6c5d02a65f8-kube-api-access-gtchn" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170436    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01ff874b-b837-480b-8adb-266dcaac4f07" volumeName="kubernetes.io/empty-dir/01ff874b-b837-480b-8adb-266dcaac4f07-tmp-dir" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170454    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="355ce6d3-5601-4133-b805-b378c9c45dd8" volumeName="kubernetes.io/projected/355ce6d3-5601-4133-b805-b378c9c45dd8-kube-api-access-cplr8" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170548    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ea0b3a-4929-431a-8cfe-410028747797" volumeName="kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-binary-copy" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170571    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351fb71-99f2-4358-bd82-895d4ffd76d9" volumeName="kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170615    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170633    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea757654-0dab-4eb1-bf75-1a38ebf1fb47" volumeName="kubernetes.io/secret/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-serving-cert" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170652    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec78fa3e-6f76-439a-b593-23572f594038" volumeName="kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170682    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ea0b3a-4929-431a-8cfe-410028747797" volumeName="kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170720    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ea0b3a-4929-431a-8cfe-410028747797" volumeName="kubernetes.io/projected/58ea0b3a-4929-431a-8cfe-410028747797-kube-api-access-tdwzx" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170741    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6170d15e-a50a-4912-bfec-932baad8ed5a" volumeName="kubernetes.io/configmap/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-config" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170765    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170778    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170795    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d77d0d5-bd4b-423f-b30a-994ad3a5e058" volumeName="kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-stats-auth" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170808    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3827824-759c-446a-b653-bed9ccf417c2" volumeName="kubernetes.io/projected/f3827824-759c-446a-b653-bed9ccf417c2-kube-api-access-2dpnz" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170828    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1df716cc-1298-4e45-bc66-f8d50e25353f" volumeName="kubernetes.io/projected/1df716cc-1298-4e45-bc66-f8d50e25353f-kube-api-access-phvvj" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170842    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3827824-759c-446a-b653-bed9ccf417c2" volumeName="kubernetes.io/secret/f3827824-759c-446a-b653-bed9ccf417c2-serving-cert" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170856    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="125530fc-c722-456a-8e21-5cae6029db06" volumeName="kubernetes.io/projected/125530fc-c722-456a-8e21-5cae6029db06-kube-api-access-djqrs" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170872    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="797abf7a-7927-468d-8a1b-c994b74224db" volumeName="kubernetes.io/projected/797abf7a-7927-468d-8a1b-c994b74224db-kube-api-access-8wjhm" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170886    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d078cc9-6850-4884-b8d2-b7e5b87f087e" volumeName="kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-snapshots" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170903    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852144f2-7df8-4279-bf86-65c5351c8945" volumeName="kubernetes.io/projected/852144f2-7df8-4279-bf86-65c5351c8945-kube-api-access-2b2pq" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170919    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4e32724-e514-41cc-b175-59da7bfd7b8b" volumeName="kubernetes.io/projected/a4e32724-e514-41cc-b175-59da7bfd7b8b-kube-api-access-6hhdj" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170934    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55ee9f5d-10b0-4ae3-946b-5295148f2429" volumeName="kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-daemon-config" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170951    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1df8e2-38c0-4e61-a117-f3b797ec963b" volumeName="kubernetes.io/projected/5e1df8e2-38c0-4e61-a117-f3b797ec963b-kube-api-access-ncmgr" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.170982    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d77d0d5-bd4b-423f-b30a-994ad3a5e058" volumeName="kubernetes.io/projected/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-kube-api-access-djdjh" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171001    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d116bb71-10c3-40c9-ac2d-50a79bc72946" volumeName="kubernetes.io/projected/d116bb71-10c3-40c9-ac2d-50a79bc72946-kube-api-access-wg7dn" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171015    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="125530fc-c722-456a-8e21-5cae6029db06" volumeName="kubernetes.io/empty-dir/125530fc-c722-456a-8e21-5cae6029db06-data-volume" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171035    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2afcc4bb-db5b-431a-bdb9-580af2eccd18" volumeName="kubernetes.io/secret/2afcc4bb-db5b-431a-bdb9-580af2eccd18-agent-certs" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171049    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db8c20d-d878-475b-97d0-6cd01befd3ae" volumeName="kubernetes.io/projected/8db8c20d-d878-475b-97d0-6cd01befd3ae-kube-api-access-jkbgm" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171066    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="965c39c6-9a97-4d32-a681-7c34eddf0f4f" volumeName="kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171082    4955 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d116bb71-10c3-40c9-ac2d-50a79bc72946" volumeName="kubernetes.io/secret/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 15:08:05.171072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171094    4955 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 15:08:05.172500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.171102    4955 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 15:08:05.177320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.177289    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/1.log"
Nov 03 15:08:05.179279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.179255    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/1.log"
Nov 03 15:08:05.190771 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.190734    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:08:05.193352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.193295    4955 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 15:08:05.193716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.193694    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:08:05.197547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.197519    4955 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 15:08:05.197547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.197549    4955 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 15:08:05.197707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.197575    4955 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 15:08:05.197707 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.197617    4955 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 15:08:05.215742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.215688    4955 generic.go:356] "Generic (PLEG): container finished" podID="daa380c87733219694504b47fb5e04ca" containerID="c988ead8bb5284ee601ce22ee10878ef6bcd50be2e27f5b598d52104135c5ee2" exitCode=0
Nov 03 15:08:05.223702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.223679    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/1.log"
Nov 03 15:08:05.223858 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.223708    4955 generic.go:356] "Generic (PLEG): container finished" podID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc" exitCode=255
Nov 03 15:08:05.235072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.235043    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:08:05.235452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.235424    4955 generic.go:356] "Generic (PLEG): container finished" podID="d116bb71-10c3-40c9-ac2d-50a79bc72946" containerID="c635da4725b69cb518362bb55969e9bc1026dda34d704a305b4f9d421860eb14" exitCode=1
Nov 03 15:08:05.245018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.244957    4955 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="170bb9a849a8468475fa09475d938e11a1eb4e40f2deb49815dfff0a770552ff" exitCode=0
Nov 03 15:08:05.245018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.245015    4955 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="bbf88af498c4fbbc07479c6944de5fb3977b64099a428c23de6a90d159064436" exitCode=0
Nov 03 15:08:05.245190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.245026    4955 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="eff950eb491ace9ea2abe8db376b8057805a9fa93d7e893a0e83f41c480d665a" exitCode=0
Nov 03 15:08:05.245190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.245035    4955 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="148a028f65e5a5e2962d32cf690a994da39d6882071a1d8f2e6ef1312deba648" exitCode=0
Nov 03 15:08:05.245190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.245045    4955 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="7289a25436bab7b6629770039b321d7aacabb7782188fcf8c5177473db1b64c6" exitCode=0
Nov 03 15:08:05.245190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.245054    4955 generic.go:356] "Generic (PLEG): container finished" podID="58ea0b3a-4929-431a-8cfe-410028747797" containerID="94c6febcca2330dde31dd74cf98684342efd9f1794e184c08c277d09e801d856" exitCode=0
Nov 03 15:08:05.259945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.259887    4955 manager.go:324] Recovery completed
Nov 03 15:08:05.264766 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.264745    4955 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found"
Nov 03 15:08:05.277586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.277565    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.278900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.278884    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.278953 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.278918    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.278953 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.278933    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.281327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281306    4955 cpu_manager.go:226] "Starting" policy="none"
Nov 03 15:08:05.281327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281325    4955 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 15:08:05.281449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281343    4955 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 15:08:05.281449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281316    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.281514 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281497    4955 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 15:08:05.281551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281516    4955 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 15:08:05.281551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281542    4955 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 15:08:05.281608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281551    4955 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 15:08:05.281608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281570    4955 policy_none.go:50] "Start"
Nov 03 15:08:05.281608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281586    4955 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 15:08:05.281608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281596    4955 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 15:08:05.281747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281736    4955 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 15:08:05.281779 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281748    4955 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 15:08:05.281779 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.281754    4955 policy_none.go:44] "Start"
Nov 03 15:08:05.282370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.282347    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.282440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.282375    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.282440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.282388    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.286809 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.286771    4955 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 15:08:05.289346 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.289331    4955 manager.go:341] "Starting Device Plugin manager"
Nov 03 15:08:05.289445 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.289356    4955 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 15:08:05.289445 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.289365    4955 server.go:87] "Starting device plugin registration server"
Nov 03 15:08:05.289898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.289846    4955 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 15:08:05.289898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.289874    4955 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 15:08:05.290086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.290070    4955 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 15:08:05.290365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.290322    4955 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 15:08:05.290486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.290476    4955 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 15:08:05.290575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.290566    4955 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 15:08:05.291416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.291299    4955 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T15:08:05.290456556Z","UUID":"75370dd2-f701-4fcd-bf40-e699bddb6490","Handler":null,"Name":"","Endpoint":""}
Nov 03 15:08:05.293172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.293153    4955 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 15:08:05.293270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.293191    4955 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 15:08:05.294045 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.293989    4955 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 15:08:05.294045 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.294029    4955 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-23.us-west-1.compute.internal\" not found"
Nov 03 15:08:05.297698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.297667    4955 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"]
Nov 03 15:08:05.297824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.297810    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.298783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.298765    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.298872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.298791    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.298872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.298800    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.299123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.299110    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.299264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.299251    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.300147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300129    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.300250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300153    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.300250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300162    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.300250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300182    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.300250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300209    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.300250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300224    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.300475 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.300294    4955 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.300608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300577    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"daa380c87733219694504b47fb5e04ca","Type":"ContainerStarted","Data":"499ca2a7909923ef00a925a8f00dbe6e82078353c3264992b1a8951b66460ceb"}
Nov 03 15:08:05.300667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300621    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"daa380c87733219694504b47fb5e04ca","Type":"ContainerDied","Data":"c988ead8bb5284ee601ce22ee10878ef6bcd50be2e27f5b598d52104135c5ee2"}
Nov 03 15:08:05.300667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300644    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"daa380c87733219694504b47fb5e04ca","Type":"ContainerStarted","Data":"3467489b98251c4ccb2dfce8777edd2995d6697c463ee5bba8794ee7a3ec07ef"}
Nov 03 15:08:05.300764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300682    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.300764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300748    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"6cc7811c328f411b1828c6a10a8039cb","Type":"ContainerStarted","Data":"3a0bb60eed91caa39ee80d4541fef0f36f03517f1b628eb482623635b6ea3647"}
Nov 03 15:08:05.300849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.300768    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal" event={"ID":"6cc7811c328f411b1828c6a10a8039cb","Type":"ContainerStarted","Data":"b30b6a09896114a41acb1a647b64fc71d56733ea3a173911af2879e3b82800e6"}
Nov 03 15:08:05.301411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.301394    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.301503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.301419    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.301503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.301429    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.301601 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.301548    4955 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-23.us-west-1.compute.internal\" not found" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.354574 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.354540    4955 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:08:05.354574 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.354572    4955 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 15:08:05.356375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.356355    4955 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:08:05.390504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.390475    4955 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 15:08:05.391638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.391620    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 15:08:05.391772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.391658    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 15:08:05.391772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.391673    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 15:08:05.391772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.391725    4955 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.405200 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.405167    4955 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:08:05.408315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.408295    4955 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 15:08:05.455217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.455189    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.455350 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.455224    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.455350 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.455258    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc7811c328f411b1828c6a10a8039cb-config\") pod \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"6cc7811c328f411b1828c6a10a8039cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.465339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.465315    4955 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.506747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.506715    4955 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.506866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.506788    4955 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.507357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.507336    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeNotReady"
Nov 03 15:08:05.507455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.507377    4955 setters.go:546] "Node became not ready" node="ip-10-0-134-23.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T15:08:05Z","lastTransitionTime":"2026-11-03T15:08:05Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 15:08:05.526057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.525962    4955 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-23.us-west-1.compute.internal" is owned by different node (oldNodeID="af03d808-afab-4a0c-ac10-c339602835aa", newNodeID=""), cleaning up...
Nov 03 15:08:05.533842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.533822    4955 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 15:08:05.533959 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.533872    4955 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.533959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.533888    4955 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.556055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.556019    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.556167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.556061    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc7811c328f411b1828c6a10a8039cb-config\") pod \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"6cc7811c328f411b1828c6a10a8039cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.556167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.556085    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.556167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.556133    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc7811c328f411b1828c6a10a8039cb-config\") pod \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"6cc7811c328f411b1828c6a10a8039cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.556268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.556134    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.556268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.556133    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa380c87733219694504b47fb5e04ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" (UID: \"daa380c87733219694504b47fb5e04ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:05.632435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:05.632407    4955 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 15:08:05.632608 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:05.632465    4955 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:06.142304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.142275    4955 apiserver.go:52] "Watching apiserver"
Nov 03 15:08:06.250922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.250889    4955 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:06.251452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.251432    4955 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:06.252084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.252062    4955 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 15:08:06.253501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.253467    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-j87d4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl","openshift-dns/dns-default-6vv5d","openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j","openshift-network-diagnostics/network-check-target-8lrk7","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt","kube-system/konnectivity-agent-pq4lr","openshift-multus/multus-additional-cni-plugins-vdz4p","kube-system/global-pull-secret-syncer-xnsmp","openshift-cluster-node-tuning-operator/tuned-8rfqr","openshift-multus/multus-fgwcf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal","openshift-dns/node-resolver-9gcrg","openshift-insights/insights-operator-7c4986d449-w7zh5","openshift-ovn-kubernetes/ovnkube-node-kzzmg","openshift-image-registry/image-registry-644965599b-fltb2","openshift-image-registry/image-registry-64c988bcf7-rdfnp","openshift-network-console/networking-console-plugin-86c478bb88-592hq","openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6","openshift-multus/network-metrics-daemon-ltnrw","kube-system/kas-connection-checker-685895c7c9-qb946","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74","openshift-insights/insights-runtime-extractor-fnprj","openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp","openshift-console-operator/console-operator-7cd56b8dbc-x66cq","openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp","openshift-network-operator/iptables-alerter-zz979","openshift-ingress/router-default-6b6f864dcc-6c9p5","kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-kz2vz","kube-system/kas-connection-checker-685895c7c9-gh68l","kube-system/kas-connection-checker-685895c7c9-lxnsf"]
Nov 03 15:08:06.254698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.254680    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.254957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.254944    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.255267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.255244    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.256552 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.256539    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.257888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.257870    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:06.260372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.260354    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:06.260883 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.260870    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.265444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.265424    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.265992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.265958    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.315583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315523    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 15:08:06.315583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315542    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 15:08:06.315583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315524    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 15:08:06.315583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315524    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315617    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315625    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315632    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315652    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315694    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315707    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315740    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315749    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.315880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.315760    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.316226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316108    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 15:08:06.316226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316205    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.316226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316212    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316246    4955 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316248    4955 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316273    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316277    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316288    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316291    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316334    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.316375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316375    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 15:08:06.316688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316521    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.316688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.316536    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 15:08:06.316688 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.316571    4955 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:06.316688 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.316575    4955 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal"
Nov 03 15:08:06.351663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.351637    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 15:08:06.353243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.353224    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 15:08:06.353893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.353876    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 15:08:06.354281 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.354266    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.354357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.354344    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.354477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.354462    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.354562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.354550    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.354891 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.354874    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.361191 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361173    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-systemd-units\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.361245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361202    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.361245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361225    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce6d3-5601-4133-b805-b378c9c45dd8-config-volume\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.361245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361241    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-bin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.361354 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361326    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3827824-759c-446a-b653-bed9ccf417c2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:08:06.361386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361352    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.361450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361436    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.361488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361461    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.361526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361488    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-config\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.361556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361534    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3827824-759c-446a-b653-bed9ccf417c2-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:08:06.361599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361536    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.361630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361593    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-system-cni-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.361630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361614    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwzx\" (UniqueName: \"kubernetes.io/projected/58ea0b3a-4929-431a-8cfe-410028747797-kube-api-access-tdwzx\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.361749 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361733    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-etc-kubernetes\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.361793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361772    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxn8v\" (UniqueName: \"kubernetes.io/projected/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-kube-api-access-mxn8v\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:08:06.361826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361812    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.361860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361841    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp85d\" (UniqueName: \"kubernetes.io/projected/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-kube-api-access-pp85d\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:08:06.361893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361870    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-modprobe-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.361930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361897    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-serving-cert\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:08:06.362004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.361962    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-trusted-ca\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.362046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362024    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.362094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362047    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.362094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362067    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbgm\" (UniqueName: \"kubernetes.io/projected/8db8c20d-d878-475b-97d0-6cd01befd3ae-kube-api-access-jkbgm\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.362094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362083    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.362094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362082    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-serving-cert\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:08:06.362237 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362146    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-conf-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.362275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362248    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-cert\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:08:06.362304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362287    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-systemd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.362341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362315    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-ovn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.362341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362335    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-script-lib\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.362397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362357    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7dn\" (UniqueName: \"kubernetes.io/projected/d116bb71-10c3-40c9-ac2d-50a79bc72946-kube-api-access-wg7dn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.362452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362439    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-lib-modules\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.362504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362487    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-host-slash\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.362551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362539    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqx28\" (UniqueName: \"kubernetes.io/projected/c6171645-b2a3-4bf8-8258-a5aef22950e7-kube-api-access-jqx28\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.362591 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362584    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6krj\" (UniqueName: \"kubernetes.io/projected/273b804d-46fc-4201-b6d1-1029b44b301d-kube-api-access-r6krj\") pod \"network-check-source-6f78d59f99-qmkmp\" (UID: \"273b804d-46fc-4201-b6d1-1029b44b301d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:08:06.362678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362667    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-cni-binary-copy\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.362789 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362775    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3827824-759c-446a-b653-bed9ccf417c2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:08:06.362830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362800    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-tmp\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.362830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362820    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-kubernetes\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.362894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362845    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb27\" (UniqueName: \"kubernetes.io/projected/01ff874b-b837-480b-8adb-266dcaac4f07-kube-api-access-7gb27\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.362894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362871    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:06.362960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362905    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-tmp\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.362960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362935    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3827824-759c-446a-b653-bed9ccf417c2-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:08:06.363043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.362997    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhdj\" (UniqueName: \"kubernetes.io/projected/a4e32724-e514-41cc-b175-59da7bfd7b8b-kube-api-access-6hhdj\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.363043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363035    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-kubelet\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.363109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363062    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-sys\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.363109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363087    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jrl\" (UniqueName: \"kubernetes.io/projected/f94b8f15-cb42-44c9-bbdc-6c609dec30db-kube-api-access-69jrl\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.363172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363118    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-iptables-alerter-script\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.363172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363150    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.363172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363169    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-bin\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.363262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363185    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-config\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.363262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363219    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-cnibin\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.363262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363245    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.363262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363249    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-netns\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.363372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363276    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.363372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363294    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-stats-auth\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.363372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363333    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1612bb40-f413-40d9-89c2-284ddc106ca7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:06.363372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363356    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxt7\" (UniqueName: \"kubernetes.io/projected/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-kube-api-access-blxt7\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:06.363372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363373    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.363522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363441    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-stats-auth\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.363522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363467    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-netns\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.363522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363507    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-kubelet-config\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.363613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363541    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.363613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363543    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-config\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:08:06.363613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363578    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.363613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363608    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvvj\" (UniqueName: \"kubernetes.io/projected/1df716cc-1298-4e45-bc66-f8d50e25353f-kube-api-access-phvvj\") pod \"volume-data-source-validator-58c9f5f46d-nn7mt\" (UID: \"1df716cc-1298-4e45-bc66-f8d50e25353f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:08:06.363737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363695    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-config\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:08:06.363737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363706    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.363799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363740    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.363799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363763    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-hostroot\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.363799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363786    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-multus-certs\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.363902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363835    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ff874b-b837-480b-8adb-266dcaac4f07-tmp-dir\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.363902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363869    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.363902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363845    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.364090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363942    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01ff874b-b837-480b-8adb-266dcaac4f07-tmp-dir\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.364090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.363963    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysconfig\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.364090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364030    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpnz\" (UniqueName: \"kubernetes.io/projected/f3827824-759c-446a-b653-bed9ccf417c2-kube-api-access-2dpnz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:08:06.364090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364083    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:06.364222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364110    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtchn\" (UniqueName: \"kubernetes.io/projected/bc79121f-b8ff-433d-962e-e6c5d02a65f8-kube-api-access-gtchn\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.364222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364133    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.364222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364158    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364211    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364246    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-tuned\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364263    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364280    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-node-log\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364298    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364313    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-k8s-cni-cncf-io\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364349    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.364359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364356    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-tuned\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364437    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d078cc9-6850-4884-b8d2-b7e5b87f087e-serving-cert\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364467    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovn-node-metrics-cert\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364515    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364537    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-system-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364599    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2pq\" (UniqueName: \"kubernetes.io/projected/852144f2-7df8-4279-bf86-65c5351c8945-kube-api-access-2b2pq\") pod \"kas-connection-checker-685895c7c9-gh68l\" (UID: \"852144f2-7df8-4279-bf86-65c5351c8945\") " pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364635    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4e32724-e514-41cc-b175-59da7bfd7b8b-serving-cert\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364656    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-daemon-config\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364680    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-device-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.364722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364695    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364740    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364759    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6zd\" (UniqueName: \"kubernetes.io/projected/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-kube-api-access-zq6zd\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364777    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplr8\" (UniqueName: \"kubernetes.io/projected/355ce6d3-5601-4133-b805-b378c9c45dd8-kube-api-access-cplr8\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364807    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj98j\" (UniqueName: \"kubernetes.io/projected/55ee9f5d-10b0-4ae3-946b-5295148f2429-kube-api-access-zj98j\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364827    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc79121f-b8ff-433d-962e-e6c5d02a65f8-serviceca\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364844    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-log-socket\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364864    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364880    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-service-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.364897    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4k9\" (UniqueName: \"kubernetes.io/projected/3d078cc9-6850-4884-b8d2-b7e5b87f087e-kube-api-access-gz4k9\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365052    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.365085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365055    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365095    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-service-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365111    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ce6d3-5601-4133-b805-b378c9c45dd8-tmp-dir\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365154    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365170    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355ce6d3-5601-4133-b805-b378c9c45dd8-tmp-dir\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365213    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365255    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-socket-dir-parent\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365318    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-multus\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365363    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-snapshots\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365401    4955 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365400    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afcc4bb-db5b-431a-bdb9-580af2eccd18-konnectivity-ca\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365425    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bn2b\" (UniqueName: \"kubernetes.io/projected/f50eed36-43f3-4f6d-8aa0-46a148ca8177-kube-api-access-5bn2b\") pod \"kas-connection-checker-685895c7c9-qb946\" (UID: \"f50eed36-43f3-4f6d-8aa0-46a148ca8177\") " pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:08:06.365486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365446    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/5e1df8e2-38c0-4e61-a117-f3b797ec963b-kube-api-access-ncmgr\") pod \"kas-connection-checker-685895c7c9-lxnsf\" (UID: \"5e1df8e2-38c0-4e61-a117-f3b797ec963b\") " pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365492    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-tmp\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365519    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdjh\" (UniqueName: \"kubernetes.io/projected/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-kube-api-access-djdjh\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365558    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3d078cc9-6850-4884-b8d2-b7e5b87f087e-snapshots\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365588    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b8f15-cb42-44c9-bbdc-6c609dec30db-tmp\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365617    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-conf\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365643    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-var-lib-kubelet\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365685    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365715    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-kubelet\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365781    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-slash\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365826    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01ff874b-b837-480b-8adb-266dcaac4f07-hosts-file\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365848    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365874    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365895    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afcc4bb-db5b-431a-bdb9-580af2eccd18-agent-certs\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365911    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-systemd\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365925    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-host\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365939    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-var-lib-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.365955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.365955    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-env-overrides\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366011    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-cnibin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366049    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-run\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366087    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366128    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366165    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-default-certificate\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366206    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366234    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlkt\" (UniqueName: \"kubernetes.io/projected/6170d15e-a50a-4912-bfec-932baad8ed5a-kube-api-access-vwlkt\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366268    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-netd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366293    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-dbus\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366317    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366361    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366350    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366380    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-default-certificate\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366405    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc79121f-b8ff-433d-962e-e6c5d02a65f8-host\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366490    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366547    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366580    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-etc-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366594    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366608    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366628    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8db8c20d-d878-475b-97d0-6cd01befd3ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366640    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366666    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366669    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-975kv\""
Nov 03 15:08:06.366728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366712    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qsch2\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.366874    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8db8c20d-d878-475b-97d0-6cd01befd3ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367142    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z77jb\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367277    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367288    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367302    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367309    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367380    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjtnz\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367396    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367408    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367414    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367426    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 15:08:06.367513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.367430    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 15:08:06.371844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.371821    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-config\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.372039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.372021    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.372646 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.372626    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32724-e514-41cc-b175-59da7bfd7b8b-trusted-ca\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.375124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.375109    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4e32724-e514-41cc-b175-59da7bfd7b8b-serving-cert\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.376263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.376248    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-env-overrides\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.377480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.377462    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.377480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.377471    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.405849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405748    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 15:08:06.405849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405763    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.405849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405804    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 15:08:06.405849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405840    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.405849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405847    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405854    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405859    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405878    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405896    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405902    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405899    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405946    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405947    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.405956    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tqtrg\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406011    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406018    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406028    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406050    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406052    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406057    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jpxkd\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406069    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406063    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406126    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406154    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406128    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-24jgf\""
Nov 03 15:08:06.406193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406203    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406233    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cblvv\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406234    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406254    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406267    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsh2r\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406360    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406418    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ncdvq\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406433    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406450    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406457    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4wckw\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406493    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406522    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zncwb\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406522    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfz9p\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406537    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406591    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9729\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406602    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406611    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406616    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406642    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdt56\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406645    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qbvnh\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406671    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406749    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406810    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7mf6v\""
Nov 03 15:08:06.406932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.406916    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 15:08:06.407889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.407023    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cpt6\""
Nov 03 15:08:06.407889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.407526    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 15:08:06.411550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.411530    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/355ce6d3-5601-4133-b805-b378c9c45dd8-config-volume\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.412116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.412096    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.413301 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.413270    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-script-lib\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.413388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.413310    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-cert\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:08:06.413388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.413365    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-cni-binary-copy\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.413493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.413428    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovnkube-config\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.413493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.413434    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-iptables-alerter-script\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.413595 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.413545    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1612bb40-f413-40d9-89c2-284ddc106ca7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:06.414487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414470    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.414538 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414501    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.414635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414619    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d078cc9-6850-4884-b8d2-b7e5b87f087e-serving-cert\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.414725 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414702    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d078cc9-6850-4884-b8d2-b7e5b87f087e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.414782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414736    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d116bb71-10c3-40c9-ac2d-50a79bc72946-ovn-node-metrics-cert\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.414782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414748    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.414869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414854    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-daemon-config\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.415003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.414990    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-original-pull-secret\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.415125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.415112    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc79121f-b8ff-433d-962e-e6c5d02a65f8-serviceca\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.415667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.415652    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58ea0b3a-4929-431a-8cfe-410028747797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.415706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.415675    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afcc4bb-db5b-431a-bdb9-580af2eccd18-konnectivity-ca\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:08:06.416813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.416798    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.416921 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.416908    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afcc4bb-db5b-431a-bdb9-580af2eccd18-agent-certs\") pod \"konnectivity-agent-pq4lr\" (UID: \"2afcc4bb-db5b-431a-bdb9-580af2eccd18\") " pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:08:06.424809 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.424788    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxt7\" (UniqueName: \"kubernetes.io/projected/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-kube-api-access-blxt7\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:06.426807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.426789    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxn8v\" (UniqueName: \"kubernetes.io/projected/9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff-kube-api-access-mxn8v\") pod \"ingress-canary-kz2vz\" (UID: \"9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff\") " pod="openshift-ingress-canary/ingress-canary-kz2vz"
Nov 03 15:08:06.429537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.429520    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.429794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.429774    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwzx\" (UniqueName: \"kubernetes.io/projected/58ea0b3a-4929-431a-8cfe-410028747797-kube-api-access-tdwzx\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.430162 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.430143    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhdj\" (UniqueName: \"kubernetes.io/projected/a4e32724-e514-41cc-b175-59da7bfd7b8b-kube-api-access-6hhdj\") pod \"console-operator-7cd56b8dbc-x66cq\" (UID: \"a4e32724-e514-41cc-b175-59da7bfd7b8b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:06.430226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.430166    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jrl\" (UniqueName: \"kubernetes.io/projected/f94b8f15-cb42-44c9-bbdc-6c609dec30db-kube-api-access-69jrl\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.430313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.430297    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbgm\" (UniqueName: \"kubernetes.io/projected/8db8c20d-d878-475b-97d0-6cd01befd3ae-kube-api-access-jkbgm\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.434024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.434009    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqx28\" (UniqueName: \"kubernetes.io/projected/c6171645-b2a3-4bf8-8258-a5aef22950e7-kube-api-access-jqx28\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.435260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.435244    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj98j\" (UniqueName: \"kubernetes.io/projected/55ee9f5d-10b0-4ae3-946b-5295148f2429-kube-api-access-zj98j\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.440528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440224    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtchn\" (UniqueName: \"kubernetes.io/projected/bc79121f-b8ff-433d-962e-e6c5d02a65f8-kube-api-access-gtchn\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.440528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440271    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp85d\" (UniqueName: \"kubernetes.io/projected/ea757654-0dab-4eb1-bf75-1a38ebf1fb47-kube-api-access-pp85d\") pod \"service-ca-operator-544d749c8c-7mzlp\" (UID: \"ea757654-0dab-4eb1-bf75-1a38ebf1fb47\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp"
Nov 03 15:08:06.440528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440416    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmgr\" (UniqueName: \"kubernetes.io/projected/5e1df8e2-38c0-4e61-a117-f3b797ec963b-kube-api-access-ncmgr\") pod \"kas-connection-checker-685895c7c9-lxnsf\" (UID: \"5e1df8e2-38c0-4e61-a117-f3b797ec963b\") " pod="kube-system/kas-connection-checker-685895c7c9-lxnsf"
Nov 03 15:08:06.440528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440456    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpnz\" (UniqueName: \"kubernetes.io/projected/f3827824-759c-446a-b653-bed9ccf417c2-kube-api-access-2dpnz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl\" (UID: \"f3827824-759c-446a-b653-bed9ccf417c2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl"
Nov 03 15:08:06.440528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440483    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplr8\" (UniqueName: \"kubernetes.io/projected/355ce6d3-5601-4133-b805-b378c9c45dd8-kube-api-access-cplr8\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.440528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440489    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdjh\" (UniqueName: \"kubernetes.io/projected/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-kube-api-access-djdjh\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.440892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440607    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkcj\" (UniqueName: \"kubernetes.io/projected/7351fb71-99f2-4358-bd82-895d4ffd76d9-kube-api-access-dzkcj\") pod \"network-check-target-8lrk7\" (UID: \"7351fb71-99f2-4358-bd82-895d4ffd76d9\") " pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:08:06.440892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440613    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6krj\" (UniqueName: \"kubernetes.io/projected/273b804d-46fc-4201-b6d1-1029b44b301d-kube-api-access-r6krj\") pod \"network-check-source-6f78d59f99-qmkmp\" (UID: \"273b804d-46fc-4201-b6d1-1029b44b301d\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-qmkmp"
Nov 03 15:08:06.440892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440766    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.440892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440838    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2pq\" (UniqueName: \"kubernetes.io/projected/852144f2-7df8-4279-bf86-65c5351c8945-kube-api-access-2b2pq\") pod \"kas-connection-checker-685895c7c9-gh68l\" (UID: \"852144f2-7df8-4279-bf86-65c5351c8945\") " pod="kube-system/kas-connection-checker-685895c7c9-gh68l"
Nov 03 15:08:06.440892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.440882    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlkt\" (UniqueName: \"kubernetes.io/projected/6170d15e-a50a-4912-bfec-932baad8ed5a-kube-api-access-vwlkt\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.444547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.444530    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.445228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.445209    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb27\" (UniqueName: \"kubernetes.io/projected/01ff874b-b837-480b-8adb-266dcaac4f07-kube-api-access-7gb27\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.445636 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.445617    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6zd\" (UniqueName: \"kubernetes.io/projected/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-kube-api-access-zq6zd\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.446109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.446090    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bn2b\" (UniqueName: \"kubernetes.io/projected/f50eed36-43f3-4f6d-8aa0-46a148ca8177-kube-api-access-5bn2b\") pod \"kas-connection-checker-685895c7c9-qb946\" (UID: \"f50eed36-43f3-4f6d-8aa0-46a148ca8177\") " pod="kube-system/kas-connection-checker-685895c7c9-qb946"
Nov 03 15:08:06.446613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.446589    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.446743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.446724    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7dn\" (UniqueName: \"kubernetes.io/projected/d116bb71-10c3-40c9-ac2d-50a79bc72946-kube-api-access-wg7dn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.449770 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.449750    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4k9\" (UniqueName: \"kubernetes.io/projected/3d078cc9-6850-4884-b8d2-b7e5b87f087e-kube-api-access-gz4k9\") pod \"insights-operator-7c4986d449-w7zh5\" (UID: \"3d078cc9-6850-4884-b8d2-b7e5b87f087e\") " pod="openshift-insights/insights-operator-7c4986d449-w7zh5"
Nov 03 15:08:06.449842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.449790    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvvj\" (UniqueName: \"kubernetes.io/projected/1df716cc-1298-4e45-bc66-f8d50e25353f-kube-api-access-phvvj\") pod \"volume-data-source-validator-58c9f5f46d-nn7mt\" (UID: \"1df716cc-1298-4e45-bc66-f8d50e25353f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nn7mt"
Nov 03 15:08:06.466901 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.466876    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.466903    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.466927    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.466943    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-system-cni-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.467006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.466980    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-etc-kubernetes\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467006    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-modprobe-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467032    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467035    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467030    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-socket-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467048    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-system-cni-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467065    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-etc-kubernetes\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467076    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467070    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467048    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-sys-fs\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467140    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467144    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-modprobe-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467189    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467221    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-conf-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467247    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-systemd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467272    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-ovn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467296    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-registration-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467320    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-lib-modules\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467309    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-conf-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467319    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-systemd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467340    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-run-ovn\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467377    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-host-slash\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467399    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-kubernetes\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467415    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-lib-modules\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467418    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467434    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14d1e13-22f0-4ab1-b82c-e1694965e8d8-host-slash\") pod \"iptables-alerter-zz979\" (UID: \"b14d1e13-22f0-4ab1-b82c-e1694965e8d8\") " pod="openshift-network-operator/iptables-alerter-zz979"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467449    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-kubelet\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467468    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-sys\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467467    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-kubernetes\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.467489    4955 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467503    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-sys\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467502    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/125530fc-c722-456a-8e21-5cae6029db06-crio-socket\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467536    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-kubelet\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.467551    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.96753753 +0000 UTC m=+1.892355449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467576    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-bin\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467610    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-bin\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467626    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-cnibin\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467653    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-netns\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467676    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467704    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-netns\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467726    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-cnibin\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467736    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467728    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-kubelet-config\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467713    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-netns\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.467772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467768    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-netns\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467812    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-kubelet-config\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467842    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467859    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-hostroot\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467872    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-multus-certs\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467887    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysconfig\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467902    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/125530fc-c722-456a-8e21-5cae6029db06-data-volume\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467920    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467931    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-hostroot\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467942    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467953    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.467984    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysconfig\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468018    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/125530fc-c722-456a-8e21-5cae6029db06-data-volume\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468052    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468073    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-multus-certs\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468080    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468116    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468134    4955 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468139    4955 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468149    4955 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468150    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468174    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.968162661 +0000 UTC m=+1.892980553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : secret "metrics-daemon-secret" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468189    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.968179829 +0000 UTC m=+1.892997720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468230    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.968216449 +0000 UTC m=+1.893034342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468260    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-node-log\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468309    4955 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468316    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-d\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468324    4955 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468334    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-node-log\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468364    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-k8s-cni-cncf-io\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468372    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.968359682 +0000 UTC m=+1.893177608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468401    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-system-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468405    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-run-k8s-cni-cncf-io\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468429    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-device-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468448    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468468    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-log-socket\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468473    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6171645-b2a3-4bf8-8258-a5aef22950e7-device-dir\") pod \"aws-ebs-csi-driver-node-9pv74\" (UID: \"c6171645-b2a3-4bf8-8258-a5aef22950e7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pv74"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468450    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-system-cni-dir\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468493    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468503    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-log-socket\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468518    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468535    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468550    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-socket-dir-parent\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468557    4955 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468571    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-multus\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468592    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-run-ovn-kubernetes\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468600    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-conf\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468608    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-multus-socket-dir-parent\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468612    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.968597271 +0000 UTC m=+1.893415175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468640    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-multus\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468649    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-var-lib-kubelet\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468658    4955 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468666    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-kubelet\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468692    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-slash\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468699    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.968691298 +0000 UTC m=+1.893509226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468714    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-slash\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468722    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01ff874b-b837-480b-8adb-266dcaac4f07-hosts-file\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468734    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-var-lib-kubelet\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468743    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-kubelet\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468746    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468763    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-systemd\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468771    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/01ff874b-b837-480b-8adb-266dcaac4f07-hosts-file\") pod \"node-resolver-9gcrg\" (UID: \"01ff874b-b837-480b-8adb-266dcaac4f07\") " pod="openshift-dns/node-resolver-9gcrg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468777    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-host\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468800    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-sysctl-conf\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468821    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-var-lib-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468837    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-host\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468843    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-etc-systemd\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468846    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-cnibin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468864    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-var-lib-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468882    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-run\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468888    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-cnibin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.468894    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.96888003 +0000 UTC m=+1.893697938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468917    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f94b8f15-cb42-44c9-bbdc-6c609dec30db-run\") pod \"tuned-8rfqr\" (UID: \"f94b8f15-cb42-44c9-bbdc-6c609dec30db\") " pod="openshift-cluster-node-tuning-operator/tuned-8rfqr"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468938    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-netd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468956    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-dbus\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.468987    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.468942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469014    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-host-cni-netd\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469022    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e3decdb-1a61-4186-9f1d-c41cac5bb6da-dbus\") pod \"global-pull-secret-syncer-xnsmp\" (UID: \"1e3decdb-1a61-4186-9f1d-c41cac5bb6da\") " pod="kube-system/global-pull-secret-syncer-xnsmp"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469023    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc79121f-b8ff-433d-962e-e6c5d02a65f8-host\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469046    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc79121f-b8ff-433d-962e-e6c5d02a65f8-host\") pod \"node-ca-j87d4\" (UID: \"bc79121f-b8ff-433d-962e-e6c5d02a65f8\") " pod="openshift-image-registry/node-ca-j87d4"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469048    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469067    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqrs\" (UniqueName: \"kubernetes.io/projected/125530fc-c722-456a-8e21-5cae6029db06-kube-api-access-djqrs\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469027    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58ea0b3a-4929-431a-8cfe-410028747797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdz4p\" (UID: \"58ea0b3a-4929-431a-8cfe-410028747797\") " pod="openshift-multus/multus-additional-cni-plugins-vdz4p"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469087    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-etc-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.469091    4955 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.469119    4955 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469122    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjhm\" (UniqueName: \"kubernetes.io/projected/797abf7a-7927-468d-8a1b-c994b74224db-kube-api-access-8wjhm\") pod \"migrator-66df5d496b-4m6j6\" (UID: \"797abf7a-7927-468d-8a1b-c994b74224db\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469126    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-etc-openvswitch\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469149    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-systemd-units\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.469155    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:06.969143902 +0000 UTC m=+1.893961795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469176    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d116bb71-10c3-40c9-ac2d-50a79bc72946-systemd-units\") pod \"ovnkube-node-kzzmg\" (UID: \"d116bb71-10c3-40c9-ac2d-50a79bc72946\") " pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469183    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-bin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.471168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.469204    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55ee9f5d-10b0-4ae3-946b-5295148f2429-host-var-lib-cni-bin\") pod \"multus-fgwcf\" (UID: \"55ee9f5d-10b0-4ae3-946b-5295148f2429\") " pod="openshift-multus/multus-fgwcf"
Nov 03 15:08:06.533377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.533311    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqrs\" (UniqueName: \"kubernetes.io/projected/125530fc-c722-456a-8e21-5cae6029db06-kube-api-access-djqrs\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.533661 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.533643    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjhm\" (UniqueName: \"kubernetes.io/projected/797abf7a-7927-468d-8a1b-c994b74224db-kube-api-access-8wjhm\") pod \"migrator-66df5d496b-4m6j6\" (UID: \"797abf7a-7927-468d-8a1b-c994b74224db\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6"
Nov 03 15:08:06.557881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.557847    4955 scope.go:122] "RemoveContainer" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc"
Nov 03 15:08:06.569004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.568985    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:08:06.569754 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.569724    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.569869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.569841    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.569869 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.569862    4955 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:06.570036 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.569915    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.06989631 +0000 UTC m=+1.994714214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:06.570036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.570011    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/125530fc-c722-456a-8e21-5cae6029db06-crio-socket\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.570122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.570095    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/125530fc-c722-456a-8e21-5cae6029db06-crio-socket\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.570423 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.570388    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/125530fc-c722-456a-8e21-5cae6029db06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:06.974071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.973994    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:06.974071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974065    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974090    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974112    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974129    4955 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974208    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974193999 +0000 UTC m=+2.899011894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974219    4955 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974224    4955 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974236    4955 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974255    4955 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974270    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974258763 +0000 UTC m=+2.899076655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : secret "metrics-daemon-secret" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974242    4955 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974283    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974276574 +0000 UTC m=+2.899094465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974295    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974288829 +0000 UTC m=+2.899106720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974137    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974304    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974299682 +0000 UTC m=+2.899117574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:08:06.974348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974350    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974374    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974417    4955 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974430    4955 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974450    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974437627 +0000 UTC m=+2.899255552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974462    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974456762 +0000 UTC m=+2.899274653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974483    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:06.974515    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974593    4955 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974600    4955 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974621    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974606924 +0000 UTC m=+2.899424828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:08:06.974908 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:06.974642    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:07.974632542 +0000 UTC m=+2.899450439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:08:07.075949 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.075910    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:07.076146 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.076081    4955 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:07.076182 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.076152    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:08.076138504 +0000 UTC m=+3.000956418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:07.255926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.255855    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:08:07.256358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.256296    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/1.log"
Nov 03 15:08:07.256358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.256327    4955 generic.go:356] "Generic (PLEG): container finished" podID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerID="37cd43782ebfe2119e6112704a1406b8679f193b2dfb1438871e3dd4ba18de1b" exitCode=255
Nov 03 15:08:07.256438 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.256410    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" event={"ID":"a4e32724-e514-41cc-b175-59da7bfd7b8b","Type":"ContainerDied","Data":"37cd43782ebfe2119e6112704a1406b8679f193b2dfb1438871e3dd4ba18de1b"}
Nov 03 15:08:07.256482 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.256469    4955 scope.go:122] "RemoveContainer" containerID="4f819fb956c697cc70980b9dc21eb9a5fb565cedaa20d039234ade95580fbbcc"
Nov 03 15:08:07.256892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.256876    4955 scope.go:122] "RemoveContainer" containerID="37cd43782ebfe2119e6112704a1406b8679f193b2dfb1438871e3dd4ba18de1b"
Nov 03 15:08:07.257124 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.257097    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-x66cq_openshift-console-operator(a4e32724-e514-41cc-b175-59da7bfd7b8b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b"
Nov 03 15:08:07.385472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.385443    4955 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:07.961910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.961882    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:07.983926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.983887    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:07.983926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.983925    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.983950    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.984002    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984047    4955 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.984063    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984116    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984096934 +0000 UTC m=+4.908914847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984046    4955 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984177    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984166878 +0000 UTC m=+4.908984775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984182    4955 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:08:07.984210 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984196    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984186751 +0000 UTC m=+4.909004642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : secret "router-metrics-certs-default" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984222    4955 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984235    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984217339 +0000 UTC m=+4.909035244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984239    4955 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644965599b-fltb2: secret "image-registry-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984278    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls podName:ec78fa3e-6f76-439a-b593-23572f594038 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984266339 +0000 UTC m=+4.909084231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls") pod "image-registry-644965599b-fltb2" (UID: "ec78fa3e-6f76-439a-b593-23572f594038") : secret "image-registry-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.984275    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.984322    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984336    4955 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.984358    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984372    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs podName:2ede9faa-be51-4210-9fa6-1dc7ddbca38b nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984359776 +0000 UTC m=+4.909177680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs") pod "network-metrics-daemon-ltnrw" (UID: "2ede9faa-be51-4210-9fa6-1dc7ddbca38b") : secret "metrics-daemon-secret" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:07.984401    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984418    4955 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984437    4955 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984472    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls podName:355ce6d3-5601-4133-b805-b378c9c45dd8 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984461581 +0000 UTC m=+4.909279507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls") pod "dns-default-6vv5d" (UID: "355ce6d3-5601-4133-b805-b378c9c45dd8") : secret "dns-default-metrics-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984488    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984480097 +0000 UTC m=+4.909297987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984499    4955 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984507    4955 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64c988bcf7-rdfnp: secret "image-registry-tls" not found
Nov 03 15:08:07.984609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:07.984543    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls podName:965c39c6-9a97-4d32-a681-7c34eddf0f4f nodeName:}" failed. No retries permitted until 2026-11-03 15:08:09.984533358 +0000 UTC m=+4.909351249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls") pod "image-registry-64c988bcf7-rdfnp" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f") : secret "image-registry-tls" not found
Nov 03 15:08:08.085277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:08.085240    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:08.085451 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:08.085389    4955 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:08.085451 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:08.085450    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:10.085435206 +0000 UTC m=+5.010253098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:08.260801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:08.260727    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:08:08.261261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:08.261137    4955 scope.go:122] "RemoveContainer" containerID="37cd43782ebfe2119e6112704a1406b8679f193b2dfb1438871e3dd4ba18de1b"
Nov 03 15:08:08.261378 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:08.261353    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-x66cq_openshift-console-operator(a4e32724-e514-41cc-b175-59da7bfd7b8b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b"
Nov 03 15:08:08.262504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:08.262478    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4m6j6" event={"ID":"797abf7a-7927-468d-8a1b-c994b74224db","Type":"ContainerStarted","Data":"dd09a05a3a4a2e1321cf26c691291a28e1c2e749f95483e7016707c100dc2b91"}
Nov 03 15:08:08.732824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:08.732792    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9gcrg_01ff874b-b837-480b-8adb-266dcaac4f07/dns-node-resolver/0.log"
Nov 03 15:08:09.148509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:09.148472    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:09.148682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:09.148670    4955 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:08:09.933372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:09.933343    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j87d4_bc79121f-b8ff-433d-962e-e6c5d02a65f8/node-ca/0.log"
Nov 03 15:08:10.002817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002770    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:10.002817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002816    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002836    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002853    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002884    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002905    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002929    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.002959    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.002990    4955 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.003036    4955 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:10.003077 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.003068    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:14.003047761 +0000 UTC m=+8.927865670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:08:10.003571 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.003095    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls podName:8db8c20d-d878-475b-97d0-6cd01befd3ae nodeName:}" failed. No retries permitted until 2026-11-03 15:08:14.00307777 +0000 UTC m=+8.927895674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jlb5j" (UID: "8db8c20d-d878-475b-97d0-6cd01befd3ae") : secret "cluster-monitoring-operator-tls" not found
Nov 03 15:08:10.003571 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.003100    4955 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:08:10.003571 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.003114    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle podName:9d77d0d5-bd4b-423f-b30a-994ad3a5e058 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:14.003105048 +0000 UTC m=+8.927922939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle") pod "router-default-6b6f864dcc-6c9p5" (UID: "9d77d0d5-bd4b-423f-b30a-994ad3a5e058") : configmap references non-existent config key: service-ca.crt
Nov 03 15:08:10.003571 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.003040    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:10.003571 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.003147    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:14.003134422 +0000 UTC m=+8.927952328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:08:10.003571 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.003274    4955 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 15:08:10.005556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.005528    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/355ce6d3-5601-4133-b805-b378c9c45dd8-metrics-tls\") pod \"dns-default-6vv5d\" (UID: \"355ce6d3-5601-4133-b805-b378c9c45dd8\") " pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:10.005876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.005856    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"image-registry-64c988bcf7-rdfnp\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") " pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:10.005960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.005879    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"image-registry-644965599b-fltb2\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") " pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:10.006031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.006008    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ede9faa-be51-4210-9fa6-1dc7ddbca38b-metrics-certs\") pod \"network-metrics-daemon-ltnrw\" (UID: \"2ede9faa-be51-4210-9fa6-1dc7ddbca38b\") " pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:10.006132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.006117    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-metrics-certs\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:10.103499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.103463    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:10.103675 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.103615    4955 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:10.103714 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.103680    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:14.103664151 +0000 UTC m=+9.028482042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:10.137521 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.137473    4955 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pq4lr_2afcc4bb-db5b-431a-bdb9-580af2eccd18/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:10.157709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.157685    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:10.159258 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.159238    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pq4lr"
Nov 03 15:08:10.160617 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.160603    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltnrw"
Nov 03 15:08:10.161714 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.161699    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:10.165843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.165831    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:10.661619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.661582    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ltnrw"]
Nov 03 15:08:10.665284 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:10.665247    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ede9faa_be51_4210_9fa6_1dc7ddbca38b.slice/crio-51d879344bf466bb4f00e699a6ff24f7a8f47eebdc01494b027a6691f5edfac2 WatchSource:0}: Error finding container 51d879344bf466bb4f00e699a6ff24f7a8f47eebdc01494b027a6691f5edfac2: Status 404 returned error can't find the container with id 51d879344bf466bb4f00e699a6ff24f7a8f47eebdc01494b027a6691f5edfac2
Nov 03 15:08:10.667426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.667274    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64c988bcf7-rdfnp"]
Nov 03 15:08:10.670442 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:10.670423    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965c39c6_9a97_4d32_a681_7c34eddf0f4f.slice/crio-9a11cab39dd50318082ba2f897786488a5332ff36bfd8dfc77741c020aeb1297 WatchSource:0}: Error finding container 9a11cab39dd50318082ba2f897786488a5332ff36bfd8dfc77741c020aeb1297: Status 404 returned error can't find the container with id 9a11cab39dd50318082ba2f897786488a5332ff36bfd8dfc77741c020aeb1297
Nov 03 15:08:10.684812 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.684787    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vv5d"]
Nov 03 15:08:10.714545 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:10.714519    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355ce6d3_5601_4133_b805_b378c9c45dd8.slice/crio-a091bc540c607da8bd5973c41a4841e20e56bc5ea3fb751ea80a2262c2ce9135 WatchSource:0}: Error finding container a091bc540c607da8bd5973c41a4841e20e56bc5ea3fb751ea80a2262c2ce9135: Status 404 returned error can't find the container with id a091bc540c607da8bd5973c41a4841e20e56bc5ea3fb751ea80a2262c2ce9135
Nov 03 15:08:10.718199 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.718174    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644965599b-fltb2"]
Nov 03 15:08:10.721203 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:10.721179    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec78fa3e_6f76_439a_b593_23572f594038.slice/crio-4a163c1c09becffe948d2d131abcf82a04160ccc6c26488d750da84e20ad1794 WatchSource:0}: Error finding container 4a163c1c09becffe948d2d131abcf82a04160ccc6c26488d750da84e20ad1794: Status 404 returned error can't find the container with id 4a163c1c09becffe948d2d131abcf82a04160ccc6c26488d750da84e20ad1794
Nov 03 15:08:10.772575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.772544    4955 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pq4lr_2afcc4bb-db5b-431a-bdb9-580af2eccd18/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:10.822210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.822163    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kz2vz_9c6edeeb-d5df-4d05-bd14-f0d3a3d84cff/serve-healthcheck-canary/0.log"
Nov 03 15:08:10.827142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.827116    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:10.827496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.827484    4955 scope.go:122] "RemoveContainer" containerID="37cd43782ebfe2119e6112704a1406b8679f193b2dfb1438871e3dd4ba18de1b"
Nov 03 15:08:10.827680 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:10.827663    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-x66cq_openshift-console-operator(a4e32724-e514-41cc-b175-59da7bfd7b8b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b"
Nov 03 15:08:10.934758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:10.934679    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4m6j6_797abf7a-7927-468d-8a1b-c994b74224db/migrator/0.log"
Nov 03 15:08:11.028530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.028495    4955 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pq4lr_2afcc4bb-db5b-431a-bdb9-580af2eccd18/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:11.277390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.277306    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644965599b-fltb2" event={"ID":"ec78fa3e-6f76-439a-b593-23572f594038","Type":"ContainerStarted","Data":"03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15"}
Nov 03 15:08:11.277390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.277353    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644965599b-fltb2" event={"ID":"ec78fa3e-6f76-439a-b593-23572f594038","Type":"ContainerStarted","Data":"4a163c1c09becffe948d2d131abcf82a04160ccc6c26488d750da84e20ad1794"}
Nov 03 15:08:11.278548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.278522    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vv5d" event={"ID":"355ce6d3-5601-4133-b805-b378c9c45dd8","Type":"ContainerStarted","Data":"a091bc540c607da8bd5973c41a4841e20e56bc5ea3fb751ea80a2262c2ce9135"}
Nov 03 15:08:11.279378 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.279340    4955 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pq4lr_2afcc4bb-db5b-431a-bdb9-580af2eccd18/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:11.279744 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.279722    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltnrw" event={"ID":"2ede9faa-be51-4210-9fa6-1dc7ddbca38b","Type":"ContainerStarted","Data":"51d879344bf466bb4f00e699a6ff24f7a8f47eebdc01494b027a6691f5edfac2"}
Nov 03 15:08:11.281104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.281083    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp" event={"ID":"965c39c6-9a97-4d32-a681-7c34eddf0f4f","Type":"ContainerStarted","Data":"a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc"}
Nov 03 15:08:11.281185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.281118    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp" event={"ID":"965c39c6-9a97-4d32-a681-7c34eddf0f4f","Type":"ContainerStarted","Data":"9a11cab39dd50318082ba2f897786488a5332ff36bfd8dfc77741c020aeb1297"}
Nov 03 15:08:11.333817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:11.333789    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl_f3827824-759c-446a-b653-bed9ccf417c2/kube-storage-version-migrator-operator/0.log"
Nov 03 15:08:12.287492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:12.287464    4955 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:08:12.288039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:12.288017    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltnrw" event={"ID":"2ede9faa-be51-4210-9fa6-1dc7ddbca38b","Type":"ContainerStarted","Data":"27370cf3beb07644ae671bf15e14d21678189c067315f0a03bce0a4e95ac9f65"}
Nov 03 15:08:12.288127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:12.288103    4955 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:08:13.293918 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:13.293882    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vv5d" event={"ID":"355ce6d3-5601-4133-b805-b378c9c45dd8","Type":"ContainerStarted","Data":"e150e0995c02107065fca859c1841338a07f6fcf4a585f5d5d8481b0429ce801"}
Nov 03 15:08:13.294434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:13.293938    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vv5d" event={"ID":"355ce6d3-5601-4133-b805-b378c9c45dd8","Type":"ContainerStarted","Data":"cc22c028963b31d98e45be57cbf28898599ac751582dcf47f6329ac967c28034"}
Nov 03 15:08:13.294434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:13.293959    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:13.295708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:13.295686    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltnrw" event={"ID":"2ede9faa-be51-4210-9fa6-1dc7ddbca38b","Type":"ContainerStarted","Data":"203156a4964b0b4c8782a5ad5ac827722698b2885c996ba86895e1156ef28d11"}
Nov 03 15:08:14.042745 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.042709    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:14.042950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.042755    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:14.042950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.042798    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:14.042950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.042846    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:14.042950 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:14.042857    4955 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 15:08:14.042950 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:14.042919    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls podName:6170d15e-a50a-4912-bfec-932baad8ed5a nodeName:}" failed. No retries permitted until 2026-11-03 15:08:22.04290599 +0000 UTC m=+16.967723896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pjwb7" (UID: "6170d15e-a50a-4912-bfec-932baad8ed5a") : secret "samples-operator-tls" not found
Nov 03 15:08:14.042950 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:14.042934    4955 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 15:08:14.043194 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:14.043006    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert podName:1612bb40-f413-40d9-89c2-284ddc106ca7 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:22.04299091 +0000 UTC m=+16.967808820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-592hq" (UID: "1612bb40-f413-40d9-89c2-284ddc106ca7") : secret "networking-console-plugin-cert" not found
Nov 03 15:08:14.043490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.043466    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d77d0d5-bd4b-423f-b30a-994ad3a5e058-service-ca-bundle\") pod \"router-default-6b6f864dcc-6c9p5\" (UID: \"9d77d0d5-bd4b-423f-b30a-994ad3a5e058\") " pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:14.045553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.045525    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8db8c20d-d878-475b-97d0-6cd01befd3ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jlb5j\" (UID: \"8db8c20d-d878-475b-97d0-6cd01befd3ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:14.055777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.055757    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"
Nov 03 15:08:14.055878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.055862    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:14.144193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.144157    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:14.144346 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:14.144322    4955 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:14.144399 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:14.144390    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls podName:125530fc-c722-456a-8e21-5cae6029db06 nodeName:}" failed. No retries permitted until 2026-11-03 15:08:22.144374664 +0000 UTC m=+17.069192572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fnprj" (UID: "125530fc-c722-456a-8e21-5cae6029db06") : secret "insights-runtime-extractor-tls" not found
Nov 03 15:08:14.535736 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.535702    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j"]
Nov 03 15:08:14.538637 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:14.538608    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db8c20d_d878_475b_97d0_6cd01befd3ae.slice/crio-63eb49fd93d19c21b2f66272946da7c69bbbb46b571b3aac6d133326300c4814 WatchSource:0}: Error finding container 63eb49fd93d19c21b2f66272946da7c69bbbb46b571b3aac6d133326300c4814: Status 404 returned error can't find the container with id 63eb49fd93d19c21b2f66272946da7c69bbbb46b571b3aac6d133326300c4814
Nov 03 15:08:14.542846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:14.542822    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b6f864dcc-6c9p5"]
Nov 03 15:08:14.588222 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:14.588194    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d77d0d5_bd4b_423f_b30a_994ad3a5e058.slice/crio-2a540b7e640b5565c39c5b47c31903bb6c08e147e053b49d985d2d09d3354025 WatchSource:0}: Error finding container 2a540b7e640b5565c39c5b47c31903bb6c08e147e053b49d985d2d09d3354025: Status 404 returned error can't find the container with id 2a540b7e640b5565c39c5b47c31903bb6c08e147e053b49d985d2d09d3354025
Nov 03 15:08:15.210390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:15.210365    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:08:15.303799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:15.303761    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5" event={"ID":"9d77d0d5-bd4b-423f-b30a-994ad3a5e058","Type":"ContainerStarted","Data":"5811e5656677ebba66248fac8fa37eb1da81304e23d2b9d859783a2b71da0648"}
Nov 03 15:08:15.304055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:15.303817    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5" event={"ID":"9d77d0d5-bd4b-423f-b30a-994ad3a5e058","Type":"ContainerStarted","Data":"2a540b7e640b5565c39c5b47c31903bb6c08e147e053b49d985d2d09d3354025"}
Nov 03 15:08:15.305166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:15.305135    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j" event={"ID":"8db8c20d-d878-475b-97d0-6cd01befd3ae","Type":"ContainerStarted","Data":"63eb49fd93d19c21b2f66272946da7c69bbbb46b571b3aac6d133326300c4814"}
Nov 03 15:08:15.743788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:15.743757    4955 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-23.us-west-1.compute.internal" event="NodeReady"
Nov 03 15:08:16.056643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:16.056567    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:16.059321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:16.059297    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:16.117148 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:16.117116    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:16.656402 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:16.656371    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:17.203343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.203275    4955 scope.go:122] "RemoveContainer" containerID="37cd43782ebfe2119e6112704a1406b8679f193b2dfb1438871e3dd4ba18de1b"
Nov 03 15:08:17.314693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.314670    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:08:17.314871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.314786    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" event={"ID":"a4e32724-e514-41cc-b175-59da7bfd7b8b","Type":"ContainerStarted","Data":"810073c30789a391fa64c9d34d6b2fad851a78516ceb938ad82a1ec57f27d3fd"}
Nov 03 15:08:17.315138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.315088    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:17.316304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.316265    4955 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-x66cq container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body=
Nov 03 15:08:17.316421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.316333    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused"
Nov 03 15:08:17.316421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.316383    4955 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:08:17.316421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.316384    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jlb5j" event={"ID":"8db8c20d-d878-475b-97d0-6cd01befd3ae","Type":"ContainerStarted","Data":"5b2d03cb127ac4bdb3ca973df7b0e65c882e50a26f3d1922f226538e575186b2"}
Nov 03 15:08:17.386777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.386739    4955 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-x66cq container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.11:8443/healthz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body=
Nov 03 15:08:17.386921 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:17.386796    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/healthz\": dial tcp 10.132.0.11:8443: connect: connection refused"
Nov 03 15:08:18.323761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:18.323733    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq"
Nov 03 15:08:19.105364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:19.105325    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:19.105573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:19.105542    4955 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 15:08:19.106558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:19.106536    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b6f864dcc-6c9p5"
Nov 03 15:08:22.118640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.118592    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:22.119079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.118657    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:22.121325 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.121296    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1612bb40-f413-40d9-89c2-284ddc106ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-592hq\" (UID: \"1612bb40-f413-40d9-89c2-284ddc106ca7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:22.121467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.121296    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6170d15e-a50a-4912-bfec-932baad8ed5a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pjwb7\" (UID: \"6170d15e-a50a-4912-bfec-932baad8ed5a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:22.156495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.156452    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"
Nov 03 15:08:22.158321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.158301    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq"
Nov 03 15:08:22.221125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.220988    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:22.223415 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.223396    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/125530fc-c722-456a-8e21-5cae6029db06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fnprj\" (UID: \"125530fc-c722-456a-8e21-5cae6029db06\") " pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:22.466762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.466731    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fnprj"
Nov 03 15:08:22.535346 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.535315    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-592hq"]
Nov 03 15:08:22.538852 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:22.538823    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1612bb40_f413_40d9_89c2_284ddc106ca7.slice/crio-9821ec22a6c7d706cd0f5b80696ad4efd7093d21b04d3950137d0b58c2d2971c WatchSource:0}: Error finding container 9821ec22a6c7d706cd0f5b80696ad4efd7093d21b04d3950137d0b58c2d2971c: Status 404 returned error can't find the container with id 9821ec22a6c7d706cd0f5b80696ad4efd7093d21b04d3950137d0b58c2d2971c
Nov 03 15:08:22.588249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.588218    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7"]
Nov 03 15:08:22.838050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:22.837959    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fnprj"]
Nov 03 15:08:22.841105 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:22.841082    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125530fc_c722_456a_8e21_5cae6029db06.slice/crio-08a3bdd4da335aeb9606faae37fedf550f12617e2e488734a5633f78fa352a52 WatchSource:0}: Error finding container 08a3bdd4da335aeb9606faae37fedf550f12617e2e488734a5633f78fa352a52: Status 404 returned error can't find the container with id 08a3bdd4da335aeb9606faae37fedf550f12617e2e488734a5633f78fa352a52
Nov 03 15:08:23.301057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:23.301029    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vv5d"
Nov 03 15:08:23.338805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:23.338772    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnprj" event={"ID":"125530fc-c722-456a-8e21-5cae6029db06","Type":"ContainerStarted","Data":"b70cfb304c1c5418021e92850b229245a7e51d0ffb95608e22de4af54d3d2afc"}
Nov 03 15:08:23.339010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:23.338835    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnprj" event={"ID":"125530fc-c722-456a-8e21-5cae6029db06","Type":"ContainerStarted","Data":"08a3bdd4da335aeb9606faae37fedf550f12617e2e488734a5633f78fa352a52"}
Nov 03 15:08:23.339959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:23.339926    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7" event={"ID":"6170d15e-a50a-4912-bfec-932baad8ed5a","Type":"ContainerStarted","Data":"a845e9d0d4538644ededc5a492a2051ba58d29e61a0469d4f430e3ea64e6dae2"}
Nov 03 15:08:23.341063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:23.341041    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq" event={"ID":"1612bb40-f413-40d9-89c2-284ddc106ca7","Type":"ContainerStarted","Data":"9821ec22a6c7d706cd0f5b80696ad4efd7093d21b04d3950137d0b58c2d2971c"}
Nov 03 15:08:24.348960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:24.348910    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-592hq" event={"ID":"1612bb40-f413-40d9-89c2-284ddc106ca7","Type":"ContainerStarted","Data":"88e4ebe137601e9d4fcdcd7ef8fa0d7a0b373bc4c3dbc8d6915c0a3237844620"}
Nov 03 15:08:24.351155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:24.351130    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnprj" event={"ID":"125530fc-c722-456a-8e21-5cae6029db06","Type":"ContainerStarted","Data":"71f28379954df8626eb611a743e3333d1b4e26ddd797938cc9f04420141c5188"}
Nov 03 15:08:24.384236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:24.384199    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64c988bcf7-rdfnp"]
Nov 03 15:08:24.633291 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:24.633215    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kzzmg"
Nov 03 15:08:25.206338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:25.206299    4955 patch_prober.go:29] interesting pod/image-registry-644965599b-fltb2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 15:08:25.206509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:25.206357    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-644965599b-fltb2" podUID="ec78fa3e-6f76-439a-b593-23572f594038" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:08:26.121070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:26.121027    4955 patch_prober.go:29] interesting pod/image-registry-644965599b-fltb2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 15:08:26.121441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:26.121076    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-644965599b-fltb2" podUID="ec78fa3e-6f76-439a-b593-23572f594038" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:08:26.362553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:26.362458    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fnprj" event={"ID":"125530fc-c722-456a-8e21-5cae6029db06","Type":"ContainerStarted","Data":"9a1b16c00fc90bd6b7344a6e7b40bfb3da3af01c814b1dfd630451fb05165230"}
Nov 03 15:08:26.364355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:26.364321    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7" event={"ID":"6170d15e-a50a-4912-bfec-932baad8ed5a","Type":"ContainerStarted","Data":"6f8f2281f91c9b38d96793faf6543a469c31417e5c4b46a5a506392cbd2c151f"}
Nov 03 15:08:26.364461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:26.364369    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pjwb7" event={"ID":"6170d15e-a50a-4912-bfec-932baad8ed5a","Type":"ContainerStarted","Data":"4d95c0b6050ce9f0cec06f3bdc270dac08bb49d5993802cdb3a993695b0210ab"}
Nov 03 15:08:27.937979 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:27.937925    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:08:34.390008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.389959    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:34.614740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.614706    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7m9rk"]
Nov 03 15:08:34.645846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.645767    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.703889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.703847    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 15:08:34.704086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.704061    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 15:08:34.729150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729110    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-textfile\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729160    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a519036c-be04-44c3-925f-d1bb53ccbba1-metrics-client-ca\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729238    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-root\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729274    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729340    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-sys\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729386    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729423    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-tls\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729464    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4db\" (UniqueName: \"kubernetes.io/projected/a519036c-be04-44c3-925f-d1bb53ccbba1-kube-api-access-5v4db\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.729641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.729534    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-wtmp\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.796679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.796643    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 15:08:34.796845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.796692    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kx5hh\""
Nov 03 15:08:34.796845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.796707    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 15:08:34.830584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830546    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-textfile\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830588    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a519036c-be04-44c3-925f-d1bb53ccbba1-metrics-client-ca\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830614    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-root\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830640    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830690    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-sys\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830729    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830742    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-root\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830763    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-tls\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830786    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4db\" (UniqueName: \"kubernetes.io/projected/a519036c-be04-44c3-925f-d1bb53ccbba1-kube-api-access-5v4db\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830801    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-sys\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.830822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830815    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-wtmp\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.831289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.830940    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-textfile\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.831289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.831010    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-wtmp\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.831289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.831253    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a519036c-be04-44c3-925f-d1bb53ccbba1-metrics-client-ca\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.831289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.831271    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.833204 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.833187    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.833297 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.833279    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a519036c-be04-44c3-925f-d1bb53ccbba1-node-exporter-tls\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.893472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.893429    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4db\" (UniqueName: \"kubernetes.io/projected/a519036c-be04-44c3-925f-d1bb53ccbba1-kube-api-access-5v4db\") pod \"node-exporter-7m9rk\" (UID: \"a519036c-be04-44c3-925f-d1bb53ccbba1\") " pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:34.996453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:34.996379    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7m9rk"
Nov 03 15:08:35.025862 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:08:35.025830    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda519036c_be04_44c3_925f_d1bb53ccbba1.slice/crio-90dff749b945e0d8be2aff5cfc3d13f800418e57db3c2d1d8e7a78f8b4d5ed94 WatchSource:0}: Error finding container 90dff749b945e0d8be2aff5cfc3d13f800418e57db3c2d1d8e7a78f8b4d5ed94: Status 404 returned error can't find the container with id 90dff749b945e0d8be2aff5cfc3d13f800418e57db3c2d1d8e7a78f8b4d5ed94
Nov 03 15:08:35.213369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:35.213332    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8lrk7"
Nov 03 15:08:35.401580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:35.401530    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m9rk" event={"ID":"a519036c-be04-44c3-925f-d1bb53ccbba1","Type":"ContainerStarted","Data":"90dff749b945e0d8be2aff5cfc3d13f800418e57db3c2d1d8e7a78f8b4d5ed94"}
Nov 03 15:08:36.122819 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:36.122785    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:08:37.412166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:37.412131    4955 generic.go:356] "Generic (PLEG): container finished" podID="a519036c-be04-44c3-925f-d1bb53ccbba1" containerID="cca540231c1d25621049f29a714024232e02e9e757e635b2461f71b6ab97f4fb" exitCode=0
Nov 03 15:08:37.412602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:37.412210    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m9rk" event={"ID":"a519036c-be04-44c3-925f-d1bb53ccbba1","Type":"ContainerDied","Data":"cca540231c1d25621049f29a714024232e02e9e757e635b2461f71b6ab97f4fb"}
Nov 03 15:08:38.419159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:38.419121    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m9rk" event={"ID":"a519036c-be04-44c3-925f-d1bb53ccbba1","Type":"ContainerStarted","Data":"91f16d16738faa20035a9f963043096825646a5da3407e8898e02a63e8145d13"}
Nov 03 15:08:38.419555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:38.419178    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m9rk" event={"ID":"a519036c-be04-44c3-925f-d1bb53ccbba1","Type":"ContainerStarted","Data":"a174707ff92fd11c6474430832fe69452a7576d5ed32bac0f461a2e7dc620364"}
Nov 03 15:08:38.592480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:38.592425    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7m9rk" podStartSLOduration=2.611907676 podStartE2EDuration="4.592384119s" totalImagesPullingTime="1.890765129s" totalInitContainerRuntime="89.711314ms" isStatefulPod=true podCreationTimestamp="2026-11-03 15:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:38.592226922 +0000 UTC m=+33.517044835" watchObservedRunningTime="2026-11-03 15:08:38.592384119 +0000 UTC m=+33.517202029"
Nov 03 15:08:46.578611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:46.578515    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-644965599b-fltb2"]
Nov 03 15:08:49.409388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.409324    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp" podUID="965c39c6-9a97-4d32-a681-7c34eddf0f4f" containerName="registry" containerID="cri-o://a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc" gracePeriod=30
Nov 03 15:08:49.665421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.665357    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:49.752171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.752138    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.752336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.752186    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.752336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.752226    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.754716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.754686    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:08:49.754716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.754691    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:08:49.754860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.754755    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:08:49.852881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.852850    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.853067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.852937    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.853067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853003    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.853067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853039    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.853239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853069    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") pod \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\" (UID: \"965c39c6-9a97-4d32-a681-7c34eddf0f4f\") "
Nov 03 15:08:49.853300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853272    4955 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-image-registry-private-configuration\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.853300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853292    4955 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-bound-sa-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.853402 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853306    4955 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/965c39c6-9a97-4d32-a681-7c34eddf0f4f-installation-pull-secrets\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.853453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853425    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:08:49.853453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.853431    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:08:49.855383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.855350    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:08:49.855782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.855748    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "kube-api-access-cdvbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:08:49.859356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.859317    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted" pod "965c39c6-9a97-4d32-a681-7c34eddf0f4f" (UID: "965c39c6-9a97-4d32-a681-7c34eddf0f4f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:08:49.954102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.954021    4955 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/965c39c6-9a97-4d32-a681-7c34eddf0f4f-ca-trust-extracted\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.954102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.954050    4955 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-certificates\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.954102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.954063    4955 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/965c39c6-9a97-4d32-a681-7c34eddf0f4f-trusted-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.954102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.954073    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvbs\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-kube-api-access-cdvbs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:49.954102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:49.954081    4955 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/965c39c6-9a97-4d32-a681-7c34eddf0f4f-registry-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:08:50.469404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.469369    4955 generic.go:356] "Generic (PLEG): container finished" podID="965c39c6-9a97-4d32-a681-7c34eddf0f4f" containerID="a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc" exitCode=0
Nov 03 15:08:50.469800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.469415    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp"
Nov 03 15:08:50.469800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.469429    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp" event={"ID":"965c39c6-9a97-4d32-a681-7c34eddf0f4f","Type":"ContainerDied","Data":"a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc"}
Nov 03 15:08:50.469800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.469470    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64c988bcf7-rdfnp" event={"ID":"965c39c6-9a97-4d32-a681-7c34eddf0f4f","Type":"ContainerDied","Data":"9a11cab39dd50318082ba2f897786488a5332ff36bfd8dfc77741c020aeb1297"}
Nov 03 15:08:50.469800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.469492    4955 scope.go:122] "RemoveContainer" containerID="a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc"
Nov 03 15:08:50.479080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.479042    4955 scope.go:122] "RemoveContainer" containerID="a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc"
Nov 03 15:08:50.479514 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:50.479491    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc\": container with ID starting with a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc not found: ID does not exist" containerID="a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc"
Nov 03 15:08:50.479612 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.479523    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc"} err="failed to get container status \"a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc\": rpc error: code = NotFound desc = could not find container \"a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc\": container with ID starting with a529f6f801b1068e05787e38cab0f7c287d1df8ef65ce738a3ebc6c40519f0dc not found: ID does not exist"
Nov 03 15:08:50.688898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.688814    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64c988bcf7-rdfnp"]
Nov 03 15:08:50.739572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:50.739537    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64c988bcf7-rdfnp"]
Nov 03 15:08:51.203190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:08:51.203157    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965c39c6-9a97-4d32-a681-7c34eddf0f4f" path="/var/lib/kubelet/pods/965c39c6-9a97-4d32-a681-7c34eddf0f4f/volumes"
Nov 03 15:08:57.937403 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:08:57.937364    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:09:11.602125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.602056    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-644965599b-fltb2" podUID="ec78fa3e-6f76-439a-b593-23572f594038" containerName="registry" containerID="cri-o://03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15" gracePeriod=30
Nov 03 15:09:11.855543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.855489    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:09:11.922466 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922433    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922478    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922533    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922627    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922673    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922717    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922786    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.922884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922820    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") pod \"ec78fa3e-6f76-439a-b593-23572f594038\" (UID: \"ec78fa3e-6f76-439a-b593-23572f594038\") "
Nov 03 15:09:11.923017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922899    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:09:11.923017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.922998    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:09:11.923295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.923153    4955 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-trusted-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:11.923635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.923616    4955 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec78fa3e-6f76-439a-b593-23572f594038-registry-certificates\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:11.925208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.925177    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:09:11.925389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.925358    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:09:11.925506 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.925385    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:09:11.925629 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.925606    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "kube-api-access-8wlcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:09:11.925725 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.925663    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:09:11.929916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:11.929896    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted" pod "ec78fa3e-6f76-439a-b593-23572f594038" (UID: "ec78fa3e-6f76-439a-b593-23572f594038"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:09:12.024592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.024560    4955 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-image-registry-private-configuration\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:12.024592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.024586    4955 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-bound-sa-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:12.024592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.024598    4955 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec78fa3e-6f76-439a-b593-23572f594038-installation-pull-secrets\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:12.024801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.024608    4955 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-registry-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:12.024801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.024617    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wlcf\" (UniqueName: \"kubernetes.io/projected/ec78fa3e-6f76-439a-b593-23572f594038-kube-api-access-8wlcf\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:12.024801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.024626    4955 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec78fa3e-6f76-439a-b593-23572f594038-ca-trust-extracted\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:09:12.553947 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.553909    4955 generic.go:356] "Generic (PLEG): container finished" podID="ec78fa3e-6f76-439a-b593-23572f594038" containerID="03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15" exitCode=0
Nov 03 15:09:12.553947 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.553949    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644965599b-fltb2"
Nov 03 15:09:12.554229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.553958    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644965599b-fltb2" event={"ID":"ec78fa3e-6f76-439a-b593-23572f594038","Type":"ContainerDied","Data":"03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15"}
Nov 03 15:09:12.554229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.554023    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644965599b-fltb2" event={"ID":"ec78fa3e-6f76-439a-b593-23572f594038","Type":"ContainerDied","Data":"4a163c1c09becffe948d2d131abcf82a04160ccc6c26488d750da84e20ad1794"}
Nov 03 15:09:12.554229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.554047    4955 scope.go:122] "RemoveContainer" containerID="03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15"
Nov 03 15:09:12.562743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.562721    4955 scope.go:122] "RemoveContainer" containerID="03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15"
Nov 03 15:09:12.563031 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:09:12.563011    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15\": container with ID starting with 03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15 not found: ID does not exist" containerID="03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15"
Nov 03 15:09:12.563083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.563041    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15"} err="failed to get container status \"03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15\": rpc error: code = NotFound desc = could not find container \"03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15\": container with ID starting with 03c630ac57c52e21fee6fe6b21ba5a31da0c761cfd3b262474513266b6223b15 not found: ID does not exist"
Nov 03 15:09:12.777627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.777593    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-644965599b-fltb2"]
Nov 03 15:09:12.828426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:12.828350    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-644965599b-fltb2"]
Nov 03 15:09:13.202690 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:13.202660    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec78fa3e-6f76-439a-b593-23572f594038" path="/var/lib/kubelet/pods/ec78fa3e-6f76-439a-b593-23572f594038/volumes"
Nov 03 15:09:27.937748 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:09:27.937713    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:09:35.636843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:35.636753    4955 generic.go:356] "Generic (PLEG): container finished" podID="3d078cc9-6850-4884-b8d2-b7e5b87f087e" containerID="5f1651b6e8eb5d46c4f7e57a1dd44dae1461699c2ab954d28bf14f1a3a7e20ca" exitCode=0
Nov 03 15:09:35.637332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:35.636834    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w7zh5" event={"ID":"3d078cc9-6850-4884-b8d2-b7e5b87f087e","Type":"ContainerDied","Data":"5f1651b6e8eb5d46c4f7e57a1dd44dae1461699c2ab954d28bf14f1a3a7e20ca"}
Nov 03 15:09:35.637332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:35.637296    4955 scope.go:122] "RemoveContainer" containerID="5f1651b6e8eb5d46c4f7e57a1dd44dae1461699c2ab954d28bf14f1a3a7e20ca"
Nov 03 15:09:35.638455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:35.638429    4955 generic.go:356] "Generic (PLEG): container finished" podID="ea757654-0dab-4eb1-bf75-1a38ebf1fb47" containerID="14e0c4446f8f65b570bd3b64e22caf1c7d2e0c22793d19547bd4fd032cf5add0" exitCode=0
Nov 03 15:09:35.638589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:35.638501    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp" event={"ID":"ea757654-0dab-4eb1-bf75-1a38ebf1fb47","Type":"ContainerDied","Data":"14e0c4446f8f65b570bd3b64e22caf1c7d2e0c22793d19547bd4fd032cf5add0"}
Nov 03 15:09:35.638824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:35.638808    4955 scope.go:122] "RemoveContainer" containerID="14e0c4446f8f65b570bd3b64e22caf1c7d2e0c22793d19547bd4fd032cf5add0"
Nov 03 15:09:36.644910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:36.644865    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w7zh5" event={"ID":"3d078cc9-6850-4884-b8d2-b7e5b87f087e","Type":"ContainerStarted","Data":"129b54d2c72ad578cbd5a70e495ef92b2d7a140afe37769d416e79e5614859ab"}
Nov 03 15:09:36.647338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:36.647303    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7mzlp" event={"ID":"ea757654-0dab-4eb1-bf75-1a38ebf1fb47","Type":"ContainerStarted","Data":"2f101bf96428f038567932bd1c098fac3ff5b9a0f9c9aed561bf093f649fece8"}
Nov 03 15:09:36.648883 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:36.648857    4955 generic.go:356] "Generic (PLEG): container finished" podID="f3827824-759c-446a-b653-bed9ccf417c2" containerID="38d444428fa246d7baaf872350925afbe1d7cbcb20e1aa0832bdee78d8027a21" exitCode=0
Nov 03 15:09:36.649027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:36.648960    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl" event={"ID":"f3827824-759c-446a-b653-bed9ccf417c2","Type":"ContainerDied","Data":"38d444428fa246d7baaf872350925afbe1d7cbcb20e1aa0832bdee78d8027a21"}
Nov 03 15:09:36.649296 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:36.649278    4955 scope.go:122] "RemoveContainer" containerID="38d444428fa246d7baaf872350925afbe1d7cbcb20e1aa0832bdee78d8027a21"
Nov 03 15:09:37.654562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:09:37.654529    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z6pwl" event={"ID":"f3827824-759c-446a-b653-bed9ccf417c2","Type":"ContainerStarted","Data":"407455a8961d05040cadc5707a10e2d04e63c3e22749a600be07e857fa64bbf3"}
Nov 03 15:09:57.937893 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:09:57.937851    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:10:27.938633 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:10:27.938595    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:10:57.937697 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:10:57.937656    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:11:27.937486 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:11:27.937442    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:11:57.937280 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:11:57.937237    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:12:14.713431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713390    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"]
Nov 03 15:12:14.713835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713714    4955 cpu_manager.go:414] "containerMap: removing container" podUID="ec78fa3e-6f76-439a-b593-23572f594038" containerName="registry"
Nov 03 15:12:14.713835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713724    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec78fa3e-6f76-439a-b593-23572f594038" containerName="registry"
Nov 03 15:12:14.713835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713743    4955 cpu_manager.go:414] "containerMap: removing container" podUID="965c39c6-9a97-4d32-a681-7c34eddf0f4f" containerName="registry"
Nov 03 15:12:14.713835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713748    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965c39c6-9a97-4d32-a681-7c34eddf0f4f" containerName="registry"
Nov 03 15:12:14.713835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713796    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec78fa3e-6f76-439a-b593-23572f594038" containerName="registry"
Nov 03 15:12:14.713835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.713806    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="965c39c6-9a97-4d32-a681-7c34eddf0f4f" containerName="registry"
Nov 03 15:12:14.716656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.716635    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.773614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.773586    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 15:12:14.777173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777137    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.777298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777224    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.777298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777261    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.777298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777288    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.777459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777357    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fc2c\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.777517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777456    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.777517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.777476    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.823622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.823582    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 15:12:14.823622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.823589    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 15:12:14.823852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.823628    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-th69q\""
Nov 03 15:12:14.823852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.823723    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 15:12:14.824630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.824603    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"]
Nov 03 15:12:14.878470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878430    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.878675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878484    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.878675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878519    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.878675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878549    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fc2c\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.878675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878585    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.878675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878640    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.878906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.878687    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.879477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.879320    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.881257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.881237    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.881340 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.881269    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.881340 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.881279    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.881413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.881369    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.941593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.941553    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:14.941743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:14.941724    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fc2c\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c\") pod \"istiod-openshift-gateway-7876bfcb74-fl4sl\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:15.028007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:15.027893    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:15.452824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:15.452795    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"]
Nov 03 15:12:15.455360 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:12:15.455329    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc17fca0_21f5_4b44_b840_f5f1df25e1d2.slice/crio-9e45e62cc89bc6977486360410647f13343340488039ccc225335b01548f0fb9 WatchSource:0}: Error finding container 9e45e62cc89bc6977486360410647f13343340488039ccc225335b01548f0fb9: Status 404 returned error can't find the container with id 9e45e62cc89bc6977486360410647f13343340488039ccc225335b01548f0fb9
Nov 03 15:12:16.203270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:16.203230    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" event={"ID":"dc17fca0-21f5-4b44-b840-f5f1df25e1d2","Type":"ContainerStarted","Data":"9e45e62cc89bc6977486360410647f13343340488039ccc225335b01548f0fb9"}
Nov 03 15:12:19.098214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:19.098174    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:12:19.098471 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:19.098251    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:12:19.215865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:19.215825    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" event={"ID":"dc17fca0-21f5-4b44-b840-f5f1df25e1d2","Type":"ContainerStarted","Data":"789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101"}
Nov 03 15:12:19.216052 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:19.215995    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:19.438099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:19.438038    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" podStartSLOduration=1.7979660819999999 podStartE2EDuration="5.438025526s" totalImagesPullingTime="3.640059444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:12:19.387042001 +0000 UTC m=+254.311859927" watchObservedRunningTime="2026-11-03 15:12:19.438025526 +0000 UTC m=+254.362843433"
Nov 03 15:12:20.221486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:20.221450    4955 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-fl4sl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:12:20.221862 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:20.221510    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:12:23.221262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:12:23.221108    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:12:27.937252 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:12:27.937216    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:12:57.938809 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:12:57.938768    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:13:05.208414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:05.208376    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:13:05.211579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:05.211558    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:13:05.217340 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:05.217320    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:13:05.220126 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:05.220104    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:13:08.856461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.856426    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"]
Nov 03 15:13:08.864396 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.864369    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.916016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.915985    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh584\" (UniqueName: \"kubernetes.io/projected/c331379e-4f0a-48e0-8461-3454ce21c303-kube-api-access-gh584\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.916016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.916020    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.916208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.916084    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c331379e-4f0a-48e0-8461-3454ce21c303-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.916208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.916150    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c331379e-4f0a-48e0-8461-3454ce21c303-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.916208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.916185    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.916303 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.916220    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:08.970629 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:08.970597    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"]
Nov 03 15:13:09.016984 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.016948    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.017175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.017018    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh584\" (UniqueName: \"kubernetes.io/projected/c331379e-4f0a-48e0-8461-3454ce21c303-kube-api-access-gh584\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.017175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.017036    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.017175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.017054    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c331379e-4f0a-48e0-8461-3454ce21c303-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.017175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.017081    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c331379e-4f0a-48e0-8461-3454ce21c303-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.017175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.017103    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.017470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.017452    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c331379e-4f0a-48e0-8461-3454ce21c303-manifests\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.019426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.019405    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c331379e-4f0a-48e0-8461-3454ce21c303-tmp\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.020279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.020249    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 15:13:09.020394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.020281    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 15:13:09.020394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.020287    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 15:13:09.020394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.020315    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkgmk\""
Nov 03 15:13:09.020394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.020315    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 15:13:09.020537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.020415    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 15:13:09.029502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.029479    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.029595 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.029529    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-metrics-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.031349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.031322    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c331379e-4f0a-48e0-8461-3454ce21c303-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.083450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.083417    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh584\" (UniqueName: \"kubernetes.io/projected/c331379e-4f0a-48e0-8461-3454ce21c303-kube-api-access-gh584\") pod \"opendatahub-operator-controller-manager-84b7cd6487-9b6dn\" (UID: \"c331379e-4f0a-48e0-8461-3454ce21c303\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.179356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.179320    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:09.671991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.671935    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"]
Nov 03 15:13:09.674796 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:13:09.674771    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc331379e_4f0a_48e0_8461_3454ce21c303.slice/crio-d6397a3d6a3b46444964b66bbe8ad64a2f364eb98f2df1fcbc97c2b3854668bc WatchSource:0}: Error finding container d6397a3d6a3b46444964b66bbe8ad64a2f364eb98f2df1fcbc97c2b3854668bc: Status 404 returned error can't find the container with id d6397a3d6a3b46444964b66bbe8ad64a2f364eb98f2df1fcbc97c2b3854668bc
Nov 03 15:13:09.676435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:09.676417    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:13:10.396924 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:10.396892    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn" event={"ID":"c331379e-4f0a-48e0-8461-3454ce21c303","Type":"ContainerStarted","Data":"d6397a3d6a3b46444964b66bbe8ad64a2f364eb98f2df1fcbc97c2b3854668bc"}
Nov 03 15:13:13.409959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:13.409919    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn" event={"ID":"c331379e-4f0a-48e0-8461-3454ce21c303","Type":"ContainerStarted","Data":"c38adfc0b46b494b290ac2133b5363647bf1c5a08a3c83189045acebe1ef93eb"}
Nov 03 15:13:14.414942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:14.414858    4955 generic.go:356] "Generic (PLEG): container finished" podID="c331379e-4f0a-48e0-8461-3454ce21c303" containerID="c38adfc0b46b494b290ac2133b5363647bf1c5a08a3c83189045acebe1ef93eb" exitCode=0
Nov 03 15:13:14.414942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:14.414899    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn" event={"ID":"c331379e-4f0a-48e0-8461-3454ce21c303","Type":"ContainerDied","Data":"c38adfc0b46b494b290ac2133b5363647bf1c5a08a3c83189045acebe1ef93eb"}
Nov 03 15:13:15.420610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:15.420579    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn" event={"ID":"c331379e-4f0a-48e0-8461-3454ce21c303","Type":"ContainerStarted","Data":"ddca5d8d123d44179e03d6fcec72cc7eaab2765300cb82f52ba465e45af975c1"}
Nov 03 15:13:15.421037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:15.420716    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:15.713680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:15.713626    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn" podStartSLOduration=3.002137411 podStartE2EDuration="7.713611718s" totalImagesPullingTime="4.229550181s" totalInitContainerRuntime="481.924126ms" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:15.663083624 +0000 UTC m=+310.587901537" watchObservedRunningTime="2026-11-03 15:13:15.713611718 +0000 UTC m=+310.638429625"
Nov 03 15:13:26.427310 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:26.427278    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84b7cd6487-9b6dn"
Nov 03 15:13:27.938427 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:13:27.938390    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:13:49.155014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.154951    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"]
Nov 03 15:13:49.160520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.160495    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.217455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.217405    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"]
Nov 03 15:13:49.231182 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.231151    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlhf\" (UniqueName: \"kubernetes.io/projected/508fd6a9-c488-4acc-a9f2-0ad7cd469837-kube-api-access-qnlhf\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.231348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.231212    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/508fd6a9-c488-4acc-a9f2-0ad7cd469837-tmp\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.231401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.231355    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508fd6a9-c488-4acc-a9f2-0ad7cd469837-tls-certs\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.312897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.312867    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 15:13:49.313130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.313108    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 15:13:49.313201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.313174    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mzpz9\""
Nov 03 15:13:49.332465 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.332430    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlhf\" (UniqueName: \"kubernetes.io/projected/508fd6a9-c488-4acc-a9f2-0ad7cd469837-kube-api-access-qnlhf\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.332649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.332472    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/508fd6a9-c488-4acc-a9f2-0ad7cd469837-tmp\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.332649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.332497    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508fd6a9-c488-4acc-a9f2-0ad7cd469837-tls-certs\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.335010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.334960    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/508fd6a9-c488-4acc-a9f2-0ad7cd469837-tmp\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.335273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.335231    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508fd6a9-c488-4acc-a9f2-0ad7cd469837-tls-certs\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.394791 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.394765    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlhf\" (UniqueName: \"kubernetes.io/projected/508fd6a9-c488-4acc-a9f2-0ad7cd469837-kube-api-access-qnlhf\") pod \"kube-auth-proxy-68fb7df4dc-bcxtb\" (UID: \"508fd6a9-c488-4acc-a9f2-0ad7cd469837\") " pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.472802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.472728    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"
Nov 03 15:13:49.859956 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:49.859933    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb"]
Nov 03 15:13:49.862355 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:13:49.862328    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508fd6a9_c488_4acc_a9f2_0ad7cd469837.slice/crio-d2b40838123aed6b478037f4e243f3ff9d678b71f39dc6b7b864a19e0424d320 WatchSource:0}: Error finding container d2b40838123aed6b478037f4e243f3ff9d678b71f39dc6b7b864a19e0424d320: Status 404 returned error can't find the container with id d2b40838123aed6b478037f4e243f3ff9d678b71f39dc6b7b864a19e0424d320
Nov 03 15:13:50.555276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:50.555242    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb" event={"ID":"508fd6a9-c488-4acc-a9f2-0ad7cd469837","Type":"ContainerStarted","Data":"d2b40838123aed6b478037f4e243f3ff9d678b71f39dc6b7b864a19e0424d320"}
Nov 03 15:13:53.570071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:53.570038    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb" event={"ID":"508fd6a9-c488-4acc-a9f2-0ad7cd469837","Type":"ContainerStarted","Data":"2ac24718fc9feb1ae0c9e8134294a20031a7ac791485824add2ad6b80217e896"}
Nov 03 15:13:53.688219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:13:53.688156    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68fb7df4dc-bcxtb" podStartSLOduration=1.803305551 podStartE2EDuration="4.688134744s" totalImagesPullingTime="2.884829193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:13:53.688135612 +0000 UTC m=+348.612953526" watchObservedRunningTime="2026-11-03 15:13:53.688134744 +0000 UTC m=+348.612952660"
Nov 03 15:13:57.938468 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:13:57.938429    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:14:27.938008 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:14:27.937950    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:14:29.270551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.270520    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:14:29.275682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.275664    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.334368 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.334337    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:14:29.363565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.363529    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5b2x\" (UniqueName: \"kubernetes.io/projected/131322e1-6527-4fe8-b6cb-c41284f51e0b-kube-api-access-d5b2x\") pod \"limitador-limitador-7c9b87d574-297t5\" (UID: \"131322e1-6527-4fe8-b6cb-c41284f51e0b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.363739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.363606    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/131322e1-6527-4fe8-b6cb-c41284f51e0b-config-file\") pod \"limitador-limitador-7c9b87d574-297t5\" (UID: \"131322e1-6527-4fe8-b6cb-c41284f51e0b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.383250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.383223    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 15:14:29.383250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.383236    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 15:14:29.383250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.383239    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 15:14:29.383848 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.383832    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-82fgg\""
Nov 03 15:14:29.464566 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.464536    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/131322e1-6527-4fe8-b6cb-c41284f51e0b-config-file\") pod \"limitador-limitador-7c9b87d574-297t5\" (UID: \"131322e1-6527-4fe8-b6cb-c41284f51e0b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.464753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.464590    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5b2x\" (UniqueName: \"kubernetes.io/projected/131322e1-6527-4fe8-b6cb-c41284f51e0b-kube-api-access-d5b2x\") pod \"limitador-limitador-7c9b87d574-297t5\" (UID: \"131322e1-6527-4fe8-b6cb-c41284f51e0b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.465192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.465171    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/131322e1-6527-4fe8-b6cb-c41284f51e0b-config-file\") pod \"limitador-limitador-7c9b87d574-297t5\" (UID: \"131322e1-6527-4fe8-b6cb-c41284f51e0b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.528130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.528057    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5b2x\" (UniqueName: \"kubernetes.io/projected/131322e1-6527-4fe8-b6cb-c41284f51e0b-kube-api-access-d5b2x\") pod \"limitador-limitador-7c9b87d574-297t5\" (UID: \"131322e1-6527-4fe8-b6cb-c41284f51e0b\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.588150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.588110    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:29.601111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.601079    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:14:29.959919 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:29.959884    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:14:29.962995 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:14:29.962955    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131322e1_6527_4fe8_b6cb_c41284f51e0b.slice/crio-bbe9cac63a9c86804d52c189715850f917439b774c0957721f035a1713535795 WatchSource:0}: Error finding container bbe9cac63a9c86804d52c189715850f917439b774c0957721f035a1713535795: Status 404 returned error can't find the container with id bbe9cac63a9c86804d52c189715850f917439b774c0957721f035a1713535795
Nov 03 15:14:30.715042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:30.715004    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5" event={"ID":"131322e1-6527-4fe8-b6cb-c41284f51e0b","Type":"ContainerStarted","Data":"bbe9cac63a9c86804d52c189715850f917439b774c0957721f035a1713535795"}
Nov 03 15:14:33.729015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:33.728957    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5" event={"ID":"131322e1-6527-4fe8-b6cb-c41284f51e0b","Type":"ContainerStarted","Data":"2829f39b207e9c6fff71aa62aed5c5060d82dc82be52ea3c876e71c9bb0aff8d"}
Nov 03 15:14:33.729420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:33.729066    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:33.849648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:33.849574    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5" podStartSLOduration=1.939611072 podStartE2EDuration="4.849559531s" totalImagesPullingTime="2.909948459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:14:33.848935615 +0000 UTC m=+388.773753529" watchObservedRunningTime="2026-11-03 15:14:33.849559531 +0000 UTC m=+388.774377435"
Nov 03 15:14:44.734264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:14:44.734183    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-297t5"
Nov 03 15:14:57.938928 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:14:57.938885    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:15:03.044358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.044323    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"]
Nov 03 15:15:03.047868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.047845    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.153388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.153354    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"]
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157288    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157423    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157483    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157529    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b240c106-868e-4be5-948f-a5072e182e28-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157594    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b240c106-868e-4be5-948f-a5072e182e28-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157674    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b240c106-868e-4be5-948f-a5072e182e28-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.159658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.157719    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66mz\" (UniqueName: \"kubernetes.io/projected/b240c106-868e-4be5-948f-a5072e182e28-kube-api-access-j66mz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258518    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b240c106-868e-4be5-948f-a5072e182e28-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258566    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66mz\" (UniqueName: \"kubernetes.io/projected/b240c106-868e-4be5-948f-a5072e182e28-kube-api-access-j66mz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258605    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258654    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258785    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258830    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b240c106-868e-4be5-948f-a5072e182e28-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.258931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.258876    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b240c106-868e-4be5-948f-a5072e182e28-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.259572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.259547    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b240c106-868e-4be5-948f-a5072e182e28-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.261669 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.261644    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b240c106-868e-4be5-948f-a5072e182e28-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.261781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.261667    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.261781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.261667    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.261781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.261702    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b240c106-868e-4be5-948f-a5072e182e28-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.321938 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.321874    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b240c106-868e-4be5-948f-a5072e182e28-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.326693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.326671    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66mz\" (UniqueName: \"kubernetes.io/projected/b240c106-868e-4be5-948f-a5072e182e28-kube-api-access-j66mz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmjhb\" (UID: \"b240c106-868e-4be5-948f-a5072e182e28\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.359549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.359518    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.789763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.789731    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"]
Nov 03 15:15:03.792890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:15:03.792865    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb240c106_868e_4be5_948f_a5072e182e28.slice/crio-8bb939b739574b498580a4b08fd179672098501156926f6a7a2c4c05cec4711c WatchSource:0}: Error finding container 8bb939b739574b498580a4b08fd179672098501156926f6a7a2c4c05cec4711c: Status 404 returned error can't find the container with id 8bb939b739574b498580a4b08fd179672098501156926f6a7a2c4c05cec4711c
Nov 03 15:15:03.794896 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.794864    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:15:03.794989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.794929    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:15:03.860480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.860442    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb" event={"ID":"b240c106-868e-4be5-948f-a5072e182e28","Type":"ContainerStarted","Data":"23a723a323fa2250d362677b90f50f9840cd32dc59fa640a3a77edde4fcd2701"}
Nov 03 15:15:03.860648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.860510    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb" event={"ID":"b240c106-868e-4be5-948f-a5072e182e28","Type":"ContainerStarted","Data":"8bb939b739574b498580a4b08fd179672098501156926f6a7a2c4c05cec4711c"}
Nov 03 15:15:03.860648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.860566    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:03.999389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:03.999331    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb" podStartSLOduration=1.999313405 podStartE2EDuration="1.999313405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:03.998664244 +0000 UTC m=+418.923482160" watchObservedRunningTime="2026-11-03 15:15:03.999313405 +0000 UTC m=+418.924131319"
Nov 03 15:15:04.866509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:04.866477    4955 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zmjhb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:15:04.866877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:04.866549    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb" podUID="b240c106-868e-4be5-948f-a5072e182e28" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:15:07.865814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:07.865777    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb"
Nov 03 15:15:08.041618 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.041579    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"]
Nov 03 15:15:08.041873 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.041843    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerName="discovery" containerID="cri-o://789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101" gracePeriod=30
Nov 03 15:15:08.221053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.221012    4955 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-fl4sl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/ready\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body=
Nov 03 15:15:08.221242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.221091    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.25:8080/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Nov 03 15:15:08.299554 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.299530    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:15:08.404753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.404710    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.404937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.404765    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.404937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.404804    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.404937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.404904    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.405101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.404945    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.405101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.405012    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.405101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.405047    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs\") pod \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\" (UID: \"dc17fca0-21f5-4b44-b840-f5f1df25e1d2\") "
Nov 03 15:15:08.405254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.405218    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:15:08.407567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.407534    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:15:08.407962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.407568    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:15:08.407962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.407715    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:15:08.407962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.407748    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:15:08.407962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.407819    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "kube-api-access-7fc2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:15:08.408539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.408514    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig" pod "dc17fca0-21f5-4b44-b840-f5f1df25e1d2" (UID: "dc17fca0-21f5-4b44-b840-f5f1df25e1d2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:15:08.506269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506233    4955 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-cacerts\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.506269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506261    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.506269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506272    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fc2c\" (UniqueName: \"kubernetes.io/projected/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-kube-api-access-7fc2c\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.506490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506281    4955 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-dns-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.506490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506290    4955 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-local-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.506490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506298    4955 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-kubeconfig\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.506490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.506306    4955 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dc17fca0-21f5-4b44-b840-f5f1df25e1d2-istio-csr-ca-configmap\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:15:08.676284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.676206    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7cx8f"]
Nov 03 15:15:08.676571 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.676559    4955 cpu_manager.go:414] "containerMap: removing container" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerName="discovery"
Nov 03 15:15:08.676613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.676573    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerName="discovery"
Nov 03 15:15:08.676647 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.676631    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerName="discovery"
Nov 03 15:15:08.680017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.679999    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.733964 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.733932    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 15:15:08.733964 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.733943    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 15:15:08.738473 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.738439    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7cx8f"]
Nov 03 15:15:08.751285 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.751258    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5fpjd\""
Nov 03 15:15:08.751421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.751258    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 15:15:08.809572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.809535    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjrnh\" (UniqueName: \"kubernetes.io/projected/d8c7575d-bac2-40c2-82ab-159007bfda0a-kube-api-access-wjrnh\") pod \"model-serving-api-75bd89fdfc-7cx8f\" (UID: \"d8c7575d-bac2-40c2-82ab-159007bfda0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.809743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.809665    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7575d-bac2-40c2-82ab-159007bfda0a-tls-certs\") pod \"model-serving-api-75bd89fdfc-7cx8f\" (UID: \"d8c7575d-bac2-40c2-82ab-159007bfda0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.882527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.882495    4955 generic.go:356] "Generic (PLEG): container finished" podID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" containerID="789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101" exitCode=0
Nov 03 15:15:08.883008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.882547    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"
Nov 03 15:15:08.883008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.882580    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" event={"ID":"dc17fca0-21f5-4b44-b840-f5f1df25e1d2","Type":"ContainerDied","Data":"789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101"}
Nov 03 15:15:08.883008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.882632    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl" event={"ID":"dc17fca0-21f5-4b44-b840-f5f1df25e1d2","Type":"ContainerDied","Data":"9e45e62cc89bc6977486360410647f13343340488039ccc225335b01548f0fb9"}
Nov 03 15:15:08.883008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.882654    4955 scope.go:122] "RemoveContainer" containerID="789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101"
Nov 03 15:15:08.891323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.891308    4955 scope.go:122] "RemoveContainer" containerID="789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101"
Nov 03 15:15:08.891612 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:15:08.891584    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101\": container with ID starting with 789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101 not found: ID does not exist" containerID="789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101"
Nov 03 15:15:08.891646 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.891621    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101"} err="failed to get container status \"789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101\": rpc error: code = NotFound desc = could not find container \"789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101\": container with ID starting with 789b28106496aa8eac33a2caa2ac8fe76ce3e11bff2c63620da6f39f41787101 not found: ID does not exist"
Nov 03 15:15:08.910562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.910531    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7575d-bac2-40c2-82ab-159007bfda0a-tls-certs\") pod \"model-serving-api-75bd89fdfc-7cx8f\" (UID: \"d8c7575d-bac2-40c2-82ab-159007bfda0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.910703 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.910586    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjrnh\" (UniqueName: \"kubernetes.io/projected/d8c7575d-bac2-40c2-82ab-159007bfda0a-kube-api-access-wjrnh\") pod \"model-serving-api-75bd89fdfc-7cx8f\" (UID: \"d8c7575d-bac2-40c2-82ab-159007bfda0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.913284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.913255    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7575d-bac2-40c2-82ab-159007bfda0a-tls-certs\") pod \"model-serving-api-75bd89fdfc-7cx8f\" (UID: \"d8c7575d-bac2-40c2-82ab-159007bfda0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.973581 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.973513    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjrnh\" (UniqueName: \"kubernetes.io/projected/d8c7575d-bac2-40c2-82ab-159007bfda0a-kube-api-access-wjrnh\") pod \"model-serving-api-75bd89fdfc-7cx8f\" (UID: \"d8c7575d-bac2-40c2-82ab-159007bfda0a\") " pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:08.992404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:08.992373    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:09.058100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:09.058069    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"]
Nov 03 15:15:09.063001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:09.062952    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fl4sl"]
Nov 03 15:15:09.204738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:09.204699    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc17fca0-21f5-4b44-b840-f5f1df25e1d2" path="/var/lib/kubelet/pods/dc17fca0-21f5-4b44-b840-f5f1df25e1d2/volumes"
Nov 03 15:15:09.364570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:09.364498    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7cx8f"]
Nov 03 15:15:09.367364 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:15:09.367342    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c7575d_bac2_40c2_82ab_159007bfda0a.slice/crio-ea51e9749a516c9face7ad601a6d4cfca31daa628ac6acc68f62178448af19a5 WatchSource:0}: Error finding container ea51e9749a516c9face7ad601a6d4cfca31daa628ac6acc68f62178448af19a5: Status 404 returned error can't find the container with id ea51e9749a516c9face7ad601a6d4cfca31daa628ac6acc68f62178448af19a5
Nov 03 15:15:09.888068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:09.888030    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f" event={"ID":"d8c7575d-bac2-40c2-82ab-159007bfda0a","Type":"ContainerStarted","Data":"ea51e9749a516c9face7ad601a6d4cfca31daa628ac6acc68f62178448af19a5"}
Nov 03 15:15:11.898640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:11.898602    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f" event={"ID":"d8c7575d-bac2-40c2-82ab-159007bfda0a","Type":"ContainerStarted","Data":"996c73b8a06a0975396ddf9f3548ba9ba82fa8a3574a36a71520fa81ae3bbb85"}
Nov 03 15:15:11.899112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:11.898794    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:12.029479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:12.029428    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f" podStartSLOduration=1.841978007 podStartE2EDuration="4.029415205s" totalImagesPullingTime="2.187437198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:15:12.028714578 +0000 UTC m=+426.953532490" watchObservedRunningTime="2026-11-03 15:15:12.029415205 +0000 UTC m=+426.954233118"
Nov 03 15:15:22.908325 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:22.908293    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7cx8f"
Nov 03 15:15:27.938198 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:15:27.938151    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:15:57.940397 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:15:57.940361    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:15:59.477263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.477226    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"]
Nov 03 15:15:59.480554 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.480538    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.540183 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.540145    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"]
Nov 03 15:15:59.572591 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.572549    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hlh\" (UniqueName: \"kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.572591 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.572590    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.572799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.572609    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.572799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.572726    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.641013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.640950    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:15:59.673441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.673410    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hlh\" (UniqueName: \"kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.673441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.673444    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.673627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.673464    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.673627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.673501    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.674109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.674093    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.692127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.692104    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:15:59.743096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.743017    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 15:15:59.743096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.743017    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p84nb\""
Nov 03 15:15:59.746437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.746416    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:15:59.949118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.949083    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 15:15:59.956472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:15:59.956448    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:00.193086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:00.193049    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hlh\" (UniqueName: \"kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh\") pod \"maas-api-6d669c7fcd-8f66x\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:00.392726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:00.392689    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:00.891587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:00.891562    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"]
Nov 03 15:16:00.907224 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:16:00.907188    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fa27a3_6cb3_4681_827d_2f813229ad2f.slice/crio-5f688d9d944c4180ad6f0b15f4236585c31828d9fdb798ab4c07a9569a426724 WatchSource:0}: Error finding container 5f688d9d944c4180ad6f0b15f4236585c31828d9fdb798ab4c07a9569a426724: Status 404 returned error can't find the container with id 5f688d9d944c4180ad6f0b15f4236585c31828d9fdb798ab4c07a9569a426724
Nov 03 15:16:01.100197 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:01.100165    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x" event={"ID":"57fa27a3-6cb3-4681-827d-2f813229ad2f","Type":"ContainerStarted","Data":"5f688d9d944c4180ad6f0b15f4236585c31828d9fdb798ab4c07a9569a426724"}
Nov 03 15:16:05.120058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:05.120024    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x" event={"ID":"57fa27a3-6cb3-4681-827d-2f813229ad2f","Type":"ContainerStarted","Data":"80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41"}
Nov 03 15:16:05.120590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:05.120127    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:05.226117 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:05.226091    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:16:06.011797 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:06.011740    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x" podStartSLOduration=3.454835959 podStartE2EDuration="7.011725178s" totalImagesPullingTime="3.556889219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:05.909708356 +0000 UTC m=+480.834526271" watchObservedRunningTime="2026-11-03 15:16:06.011725178 +0000 UTC m=+480.936543092"
Nov 03 15:16:11.129097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:11.129061    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:12.396856 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.396819    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"]
Nov 03 15:16:12.401683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.401660    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.447800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.447762    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"]
Nov 03 15:16:12.476804 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.476770    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.476942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.476814    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.476942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.476841    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphp5\" (UniqueName: \"kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.476942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.476924    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.578167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.578129    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.578351 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.578195    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.578351 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.578298    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.578351 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.578343    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphp5\" (UniqueName: \"kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.578827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.578800    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.580733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.580713    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.580832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.580750    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:12.862328 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:12.862259    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphp5\" (UniqueName: \"kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5\") pod \"maas-api-6c798699b4-jwtdk\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") " pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:13.013873 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:13.013832    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:13.426818 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:16:13.426786    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb728134d_dd38_443e_8c8f_6057caa22bac.slice/crio-48d4e0fc7622ea8645282cf4b9a3459dc0ba37e4d2a49141db78cc3bcdeb0412 WatchSource:0}: Error finding container 48d4e0fc7622ea8645282cf4b9a3459dc0ba37e4d2a49141db78cc3bcdeb0412: Status 404 returned error can't find the container with id 48d4e0fc7622ea8645282cf4b9a3459dc0ba37e4d2a49141db78cc3bcdeb0412
Nov 03 15:16:13.493509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:13.493475    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"]
Nov 03 15:16:14.157217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:14.157178    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk" event={"ID":"b728134d-dd38-443e-8c8f-6057caa22bac","Type":"ContainerStarted","Data":"48d4e0fc7622ea8645282cf4b9a3459dc0ba37e4d2a49141db78cc3bcdeb0412"}
Nov 03 15:16:15.162619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:15.162583    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk" event={"ID":"b728134d-dd38-443e-8c8f-6057caa22bac","Type":"ContainerStarted","Data":"7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc"}
Nov 03 15:16:15.163060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:15.162828    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:15.714562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:15.714506    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk" podStartSLOduration=2.905283386 podStartE2EDuration="3.714493351s" totalImagesPullingTime="809.209965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:15.663159142 +0000 UTC m=+490.587977054" watchObservedRunningTime="2026-11-03 15:16:15.714493351 +0000 UTC m=+490.639311332"
Nov 03 15:16:21.172534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.172506    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 15:16:21.550036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.549904    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"]
Nov 03 15:16:21.550231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.550202    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x" podUID="57fa27a3-6cb3-4681-827d-2f813229ad2f" containerName="maas-api" containerID="cri-o://80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41" gracePeriod=30
Nov 03 15:16:21.795224 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.795201    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:21.859783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.859701    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca\") pod \"57fa27a3-6cb3-4681-827d-2f813229ad2f\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") "
Nov 03 15:16:21.859783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.859746    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls\") pod \"57fa27a3-6cb3-4681-827d-2f813229ad2f\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") "
Nov 03 15:16:21.859783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.859782    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs\") pod \"57fa27a3-6cb3-4681-827d-2f813229ad2f\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") "
Nov 03 15:16:21.860061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.859897    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh\" (UniqueName: \"kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh\") pod \"57fa27a3-6cb3-4681-827d-2f813229ad2f\" (UID: \"57fa27a3-6cb3-4681-827d-2f813229ad2f\") "
Nov 03 15:16:21.860149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.860124    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca" pod "57fa27a3-6cb3-4681-827d-2f813229ad2f" (UID: "57fa27a3-6cb3-4681-827d-2f813229ad2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:16:21.860228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.860208    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fa27a3-6cb3-4681-827d-2f813229ad2f-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:16:21.862124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.862094    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls" pod "57fa27a3-6cb3-4681-827d-2f813229ad2f" (UID: "57fa27a3-6cb3-4681-827d-2f813229ad2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:16:21.862243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.862126    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs" pod "57fa27a3-6cb3-4681-827d-2f813229ad2f" (UID: "57fa27a3-6cb3-4681-827d-2f813229ad2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:16:21.862243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.862203    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh" pod "57fa27a3-6cb3-4681-827d-2f813229ad2f" (UID: "57fa27a3-6cb3-4681-827d-2f813229ad2f"). InnerVolumeSpecName "kube-api-access-r5hlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:16:21.960632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.960598    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:16:21.960632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.960625    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fa27a3-6cb3-4681-827d-2f813229ad2f-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:16:21.960632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:21.960635    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hlh\" (UniqueName: \"kubernetes.io/projected/57fa27a3-6cb3-4681-827d-2f813229ad2f-kube-api-access-r5hlh\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:16:22.192393 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.192359    4955 generic.go:356] "Generic (PLEG): container finished" podID="57fa27a3-6cb3-4681-827d-2f813229ad2f" containerID="80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41" exitCode=0
Nov 03 15:16:22.192837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.192405    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"
Nov 03 15:16:22.192837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.192445    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x" event={"ID":"57fa27a3-6cb3-4681-827d-2f813229ad2f","Type":"ContainerDied","Data":"80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41"}
Nov 03 15:16:22.192837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.192505    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x" event={"ID":"57fa27a3-6cb3-4681-827d-2f813229ad2f","Type":"ContainerDied","Data":"5f688d9d944c4180ad6f0b15f4236585c31828d9fdb798ab4c07a9569a426724"}
Nov 03 15:16:22.192837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.192533    4955 scope.go:122] "RemoveContainer" containerID="80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41"
Nov 03 15:16:22.201827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.201809    4955 scope.go:122] "RemoveContainer" containerID="80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41"
Nov 03 15:16:22.202120 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:16:22.202098    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41\": container with ID starting with 80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41 not found: ID does not exist" containerID="80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41"
Nov 03 15:16:22.202230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.202126    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41"} err="failed to get container status \"80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41\": rpc error: code = NotFound desc = could not find container \"80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41\": container with ID starting with 80f06c1d1093ba1725ee1d3f0a37ab00c05905df9150a578e785ddf63839fb41 not found: ID does not exist"
Nov 03 15:16:22.555013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.554912    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"]
Nov 03 15:16:22.759818 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:22.759783    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-8f66x"]
Nov 03 15:16:23.202953 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:16:23.202920    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fa27a3-6cb3-4681-827d-2f813229ad2f" path="/var/lib/kubelet/pods/57fa27a3-6cb3-4681-827d-2f813229ad2f/volumes"
Nov 03 15:16:27.938315 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:16:27.938277    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:16:57.937797 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:16:57.937754    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:17:16.215906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.215868    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"]
Nov 03 15:17:16.216530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.216403    4955 cpu_manager.go:414] "containerMap: removing container" podUID="57fa27a3-6cb3-4681-827d-2f813229ad2f" containerName="maas-api"
Nov 03 15:17:16.216530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.216421    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fa27a3-6cb3-4681-827d-2f813229ad2f" containerName="maas-api"
Nov 03 15:17:16.216530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.216506    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fa27a3-6cb3-4681-827d-2f813229ad2f" containerName="maas-api"
Nov 03 15:17:16.219421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.219399    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.286774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.286742    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.286928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.286789    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.286928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.286889    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4848f69-a0a9-4007-9203-7c656b86ef62-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.287067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.286957    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.287102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.287061    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwc2\" (UniqueName: \"kubernetes.io/projected/a4848f69-a0a9-4007-9203-7c656b86ef62-kube-api-access-9xwc2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.287102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.287089    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.323769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.323738    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-whvzx\""
Nov 03 15:17:16.336750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.336722    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 15:17:16.367041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.367003    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 15:17:16.375747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.375715    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"]
Nov 03 15:17:16.387828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.387796    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.387950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.387852    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.387950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.387885    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4848f69-a0a9-4007-9203-7c656b86ef62-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.387950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.387925    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.388140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.387963    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwc2\" (UniqueName: \"kubernetes.io/projected/a4848f69-a0a9-4007-9203-7c656b86ef62-kube-api-access-9xwc2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.388140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.388020    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.388276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.388255    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.388338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.388317    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.388438 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.388420    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.390304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.390286    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4848f69-a0a9-4007-9203-7c656b86ef62-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.476077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.476004    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 15:17:16.478716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.478688    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwc2\" (UniqueName: \"kubernetes.io/projected/a4848f69-a0a9-4007-9203-7c656b86ef62-kube-api-access-9xwc2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.480650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.480628    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4848f69-a0a9-4007-9203-7c656b86ef62-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4xf6\" (UID: \"a4848f69-a0a9-4007-9203-7c656b86ef62\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.531847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.531814    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:16.903613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:16.903582    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"]
Nov 03 15:17:16.957022 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:17:16.956993    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4848f69_a0a9_4007_9203_7c656b86ef62.slice/crio-565e8d000ed32d1a87c09bdb8aee5bd63af723aac4445b1679788d21ce4385a6 WatchSource:0}: Error finding container 565e8d000ed32d1a87c09bdb8aee5bd63af723aac4445b1679788d21ce4385a6: Status 404 returned error can't find the container with id 565e8d000ed32d1a87c09bdb8aee5bd63af723aac4445b1679788d21ce4385a6
Nov 03 15:17:17.421114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:17.421076    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" event={"ID":"a4848f69-a0a9-4007-9203-7c656b86ef62","Type":"ContainerStarted","Data":"565e8d000ed32d1a87c09bdb8aee5bd63af723aac4445b1679788d21ce4385a6"}
Nov 03 15:17:19.430819 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:19.430782    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" event={"ID":"a4848f69-a0a9-4007-9203-7c656b86ef62","Type":"ContainerStarted","Data":"d6d12808fbeacf19efd290eb1c54d0b0f08e7e5b82b4b1be7bdffd6a9aec52a0"}
Nov 03 15:17:19.532590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:19.532492    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:19.600817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:19.600764    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" podStartSLOduration=1.288207705 podStartE2EDuration="3.600751668s" totalImagesPullingTime="2.312543963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:19.54967588 +0000 UTC m=+554.474493787" watchObservedRunningTime="2026-11-03 15:17:19.600751668 +0000 UTC m=+554.525569581"
Nov 03 15:17:19.835206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:19.835133    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:17:20.156721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.156689    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"]
Nov 03 15:17:20.160089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.160071    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.217024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.216994    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff4535e-af16-4971-8c47-6777f461d121-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.217176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.217063    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5l7\" (UniqueName: \"kubernetes.io/projected/cff4535e-af16-4971-8c47-6777f461d121-kube-api-access-cv5l7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.217218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.217171    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.217295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.217282    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.217342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.217328    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.217407 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.217391    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.264252 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.264220    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 15:17:20.265438 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.265413    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"]
Nov 03 15:17:20.318661 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.318628    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.318846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.318666    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff4535e-af16-4971-8c47-6777f461d121-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.318846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.318804    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5l7\" (UniqueName: \"kubernetes.io/projected/cff4535e-af16-4971-8c47-6777f461d121-kube-api-access-cv5l7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.318999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.318843    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.318999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.318879    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.318999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.318930    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.319286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.319258    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.319383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.319311    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.319383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.319347    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.321551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.321528    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff4535e-af16-4971-8c47-6777f461d121-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.321793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.321774    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff4535e-af16-4971-8c47-6777f461d121-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.381472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.381434    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5l7\" (UniqueName: \"kubernetes.io/projected/cff4535e-af16-4971-8c47-6777f461d121-kube-api-access-cv5l7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6\" (UID: \"cff4535e-af16-4971-8c47-6777f461d121\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.472227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.472147    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:20.845202 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:20.845122    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"]
Nov 03 15:17:20.885045 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:17:20.885011    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff4535e_af16_4971_8c47_6777f461d121.slice/crio-875798a5b2194e35fe0265d65398614e314ef3d13cef59313c8303b8a7f78f3b WatchSource:0}: Error finding container 875798a5b2194e35fe0265d65398614e314ef3d13cef59313c8303b8a7f78f3b: Status 404 returned error can't find the container with id 875798a5b2194e35fe0265d65398614e314ef3d13cef59313c8303b8a7f78f3b
Nov 03 15:17:21.440290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:21.440253    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6" event={"ID":"cff4535e-af16-4971-8c47-6777f461d121","Type":"ContainerStarted","Data":"875798a5b2194e35fe0265d65398614e314ef3d13cef59313c8303b8a7f78f3b"}
Nov 03 15:17:22.446417 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:22.446378    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6" event={"ID":"cff4535e-af16-4971-8c47-6777f461d121","Type":"ContainerStarted","Data":"2eb22c451d4e0df2da640258d06c75e290057322c3bee850e1baee5f69fadef7"}
Nov 03 15:17:22.472720 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:22.472689    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:22.616925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:22.616872    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6" podStartSLOduration=1.863872326 podStartE2EDuration="2.616856551s" totalImagesPullingTime="752.984225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:22.566047775 +0000 UTC m=+557.490865685" watchObservedRunningTime="2026-11-03 15:17:22.616856551 +0000 UTC m=+557.541674457"
Nov 03 15:17:26.532995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:26.532942    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:26.546411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:26.546388    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:27.479957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:27.479920    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6"
Nov 03 15:17:27.938622 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:17:27.938585    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:17:28.635120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.635089    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:17:28.686195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.686156    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"]
Nov 03 15:17:28.689482 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.689462    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.761546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.761520    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 15:17:28.792551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.792519    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.792735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.792571    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.792735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.792638    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.792735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.792678    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.792735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.792708    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.792874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.792769    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqsd\" (UniqueName: \"kubernetes.io/projected/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-kube-api-access-vtqsd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.794768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.794748    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"]
Nov 03 15:17:28.893805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.893777    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.893957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.893826    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqsd\" (UniqueName: \"kubernetes.io/projected/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-kube-api-access-vtqsd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.893957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.893872    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.893957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.893901    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.894118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.894022    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.894118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.894070    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.894244 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.894224    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.894381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.894361    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.894476 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.894434    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.896254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.896233    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.896535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.896517    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:28.961956 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:28.961923    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqsd\" (UniqueName: \"kubernetes.io/projected/44c5a3f3-e5c3-4f67-99de-5eb55e05bd06-kube-api-access-vtqsd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lhbx9\" (UID: \"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:29.001818 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:29.001789    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:29.378543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:29.378455    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"]
Nov 03 15:17:29.381469 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:17:29.381439    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c5a3f3_e5c3_4f67_99de_5eb55e05bd06.slice/crio-51fac4d2d0a59efde312a7aa900f80569a36478749473707d5bc863626d3db69 WatchSource:0}: Error finding container 51fac4d2d0a59efde312a7aa900f80569a36478749473707d5bc863626d3db69: Status 404 returned error can't find the container with id 51fac4d2d0a59efde312a7aa900f80569a36478749473707d5bc863626d3db69
Nov 03 15:17:29.479248 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:29.479208    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9" event={"ID":"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06","Type":"ContainerStarted","Data":"51fac4d2d0a59efde312a7aa900f80569a36478749473707d5bc863626d3db69"}
Nov 03 15:17:30.472369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:30.472287    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:30.485515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:30.485491    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:30.486225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:30.486190    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9" event={"ID":"44c5a3f3-e5c3-4f67-99de-5eb55e05bd06","Type":"ContainerStarted","Data":"b70ee7d30dca6e835df0ef43c94d55be570270009fb21ebcddcca0beb5744856"}
Nov 03 15:17:30.497278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:30.497257    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6"
Nov 03 15:17:30.898148 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:30.898104    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9" podStartSLOduration=2.091610474 podStartE2EDuration="2.898090371s" totalImagesPullingTime="806.479897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:30.897109946 +0000 UTC m=+565.821927859" watchObservedRunningTime="2026-11-03 15:17:30.898090371 +0000 UTC m=+565.822908284"
Nov 03 15:17:31.002193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:31.002139    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:37.614579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:37.614542    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:17:39.001930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:39.001899    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:39.014195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:39.014171    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:39.536900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:39.536872    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9"
Nov 03 15:17:46.545919 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.545844    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"]
Nov 03 15:17:46.549142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.549126    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.645637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.645602    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.645794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.645651    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.645794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.645752    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.645794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.645781    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.645910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.645815    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jsx\" (UniqueName: \"kubernetes.io/projected/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-kube-api-access-l2jsx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.645910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.645869    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.652836 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.652811    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 15:17:46.654268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.654243    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"]
Nov 03 15:17:46.747201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747166    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747206    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747468 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747345    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747468 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747397    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747468 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747443    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jsx\" (UniqueName: \"kubernetes.io/projected/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-kube-api-access-l2jsx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747629 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747483    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747629 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747550    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747754 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747734    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.747802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.747751    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.749579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.749559    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.749945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.749926    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.810111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.810042    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jsx\" (UniqueName: \"kubernetes.io/projected/45c5a8e6-be22-4a5f-862e-c7d47d36f39a-kube-api-access-l2jsx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg\" (UID: \"45c5a8e6-be22-4a5f-862e-c7d47d36f39a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:46.861795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:46.861758    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:47.258632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:47.258608    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"]
Nov 03 15:17:47.261230 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:17:47.261203    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c5a8e6_be22_4a5f_862e_c7d47d36f39a.slice/crio-b5d119329fb838886cd25f14d9e446143ffff13afac828557b1f08679aba4a69 WatchSource:0}: Error finding container b5d119329fb838886cd25f14d9e446143ffff13afac828557b1f08679aba4a69: Status 404 returned error can't find the container with id b5d119329fb838886cd25f14d9e446143ffff13afac828557b1f08679aba4a69
Nov 03 15:17:47.560329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:47.560247    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg" event={"ID":"45c5a8e6-be22-4a5f-862e-c7d47d36f39a","Type":"ContainerStarted","Data":"b5d119329fb838886cd25f14d9e446143ffff13afac828557b1f08679aba4a69"}
Nov 03 15:17:48.566566 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:48.566533    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg" event={"ID":"45c5a8e6-be22-4a5f-862e-c7d47d36f39a","Type":"ContainerStarted","Data":"37a3e0685558399103944a7214dec3450fabbe05689fc1be6bca518f377d23a5"}
Nov 03 15:17:48.686989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:48.686926    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg" podStartSLOduration=1.8996912080000001 podStartE2EDuration="2.686913065s" totalImagesPullingTime="787.221857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:48.686492203 +0000 UTC m=+583.611310117" watchObservedRunningTime="2026-11-03 15:17:48.686913065 +0000 UTC m=+583.611730977"
Nov 03 15:17:48.862136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:48.862043    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:50.585048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:50.585014    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:17:51.100792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.100756    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"]
Nov 03 15:17:51.104231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.104215    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.158225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.158185    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 15:17:51.187603 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.187563    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.187780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.187683    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.187780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.187712    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.187892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.187793    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wps49\" (UniqueName: \"kubernetes.io/projected/958b5803-9348-4904-ac21-82254f5f9b90-kube-api-access-wps49\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.187928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.187889    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958b5803-9348-4904-ac21-82254f5f9b90-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.187962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.187926    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.209550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.209518    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"]
Nov 03 15:17:51.289336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289300    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289466    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289502    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289562    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wps49\" (UniqueName: \"kubernetes.io/projected/958b5803-9348-4904-ac21-82254f5f9b90-kube-api-access-wps49\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289602    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958b5803-9348-4904-ac21-82254f5f9b90-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289651    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289823 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289745    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289823 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289802    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.289908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.289823    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.292622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.292598    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958b5803-9348-4904-ac21-82254f5f9b90-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.292724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.292689    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958b5803-9348-4904-ac21-82254f5f9b90-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.353390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.353323    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wps49\" (UniqueName: \"kubernetes.io/projected/958b5803-9348-4904-ac21-82254f5f9b90-kube-api-access-wps49\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt\" (UID: \"958b5803-9348-4904-ac21-82254f5f9b90\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.428052 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.428011    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:51.805402 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:51.805374    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"]
Nov 03 15:17:51.807759 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:17:51.807737    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod958b5803_9348_4904_ac21_82254f5f9b90.slice/crio-14c57bd4d758530b60307c3c7e14b7ad1657e548cad09875abc900edc1e97350 WatchSource:0}: Error finding container 14c57bd4d758530b60307c3c7e14b7ad1657e548cad09875abc900edc1e97350: Status 404 returned error can't find the container with id 14c57bd4d758530b60307c3c7e14b7ad1657e548cad09875abc900edc1e97350
Nov 03 15:17:52.586050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:52.586014    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" event={"ID":"958b5803-9348-4904-ac21-82254f5f9b90","Type":"ContainerStarted","Data":"14c57bd4d758530b60307c3c7e14b7ad1657e548cad09875abc900edc1e97350"}
Nov 03 15:17:53.598044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:53.598007    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" event={"ID":"958b5803-9348-4904-ac21-82254f5f9b90","Type":"ContainerStarted","Data":"867f83468a9661a552e86241c307f23f1a44b70b02a0d0b125fc9898b03909e3"}
Nov 03 15:17:53.720423 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:53.720368    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podStartSLOduration=1.758808258 podStartE2EDuration="2.72035558s" totalImagesPullingTime="961.547322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:53.719333589 +0000 UTC m=+588.644151515" watchObservedRunningTime="2026-11-03 15:17:53.72035558 +0000 UTC m=+588.645173493"
Nov 03 15:17:54.303647 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.303611    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"]
Nov 03 15:17:54.307501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.307485    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.361367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.361331    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n82lg\""
Nov 03 15:17:54.414177 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.414137    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"]
Nov 03 15:17:54.426031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426004    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426035    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426054    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rzj\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426078    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426297 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426212    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426297 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426250    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426306    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426338    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.426442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.426420    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.428464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.428432    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:17:54.526963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.526926    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.526963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.526955    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.526995    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rzj\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527017    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527135    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527210    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527255    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527305    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527331    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527436    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527681    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527793    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.527874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.527851    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.528088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.528071    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.529837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.529815    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.530105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.530089    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.589793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.589723    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rzj\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.590033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.590011    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-j4d8k\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:54.621095 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:54.621064    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:17:55.048244 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:55.048171    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"]
Nov 03 15:17:55.050840 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:17:55.050809    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e12184_fe13_4680_8917_4054a8bf528e.slice/crio-84bf42beeda3be21dd7f279c8e62073d59f503b66a7357e28d08efd0926862db WatchSource:0}: Error finding container 84bf42beeda3be21dd7f279c8e62073d59f503b66a7357e28d08efd0926862db: Status 404 returned error can't find the container with id 84bf42beeda3be21dd7f279c8e62073d59f503b66a7357e28d08efd0926862db
Nov 03 15:17:55.608908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:55.608867    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" event={"ID":"e1e12184-fe13-4680-8917-4054a8bf528e","Type":"ContainerStarted","Data":"84bf42beeda3be21dd7f279c8e62073d59f503b66a7357e28d08efd0926862db"}
Nov 03 15:17:56.862755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:56.862717    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:56.879140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:56.879108    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:57.634348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:57.634263    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg"
Nov 03 15:17:57.942603 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:17:57.942565    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:17:58.773469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:58.773432    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:17:58.773590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:58.773505    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:17:58.773590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:58.773538    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:17:59.655813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:59.655772    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" event={"ID":"e1e12184-fe13-4680-8917-4054a8bf528e","Type":"ContainerStarted","Data":"1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6"}
Nov 03 15:17:59.840998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:17:59.829954    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podStartSLOduration=2.11020403 podStartE2EDuration="5.829935014s" totalImagesPullingTime="3.719730984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:59.778177451 +0000 UTC m=+594.702995365" watchObservedRunningTime="2026-11-03 15:17:59.829935014 +0000 UTC m=+594.754752930"
Nov 03 15:18:00.621960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:00.621926    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:18:00.623515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:00.623487    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-j4d8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 15:18:00.623657 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:00.623557    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 15:18:01.428458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:01.428417    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:18:01.465037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:01.465010    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:18:01.622103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:01.622066    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-j4d8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 15:18:01.622283 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:01.622119    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 15:18:01.675508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:01.675479    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:18:02.621395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:02.621362    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-j4d8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 15:18:02.621826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:02.621424    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 15:18:03.622228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:03.622186    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-j4d8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 15:18:03.622730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:03.622240    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 15:18:04.095087 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:04.095052    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:18:04.621452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:04.621409    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-j4d8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 15:18:04.621630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:04.621472    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 15:18:05.238206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:05.238180    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:18:05.238600 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:05.238295    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:18:05.247328 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:05.247302    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:18:05.247553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:05.247534    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:18:05.622206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:05.622116    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-j4d8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 03 15:18:05.622361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:05.622199    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 03 15:18:07.228522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:07.228492    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:18:07.229042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:07.228800    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:18:07.229672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:07.229648    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:18:24.321786 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.321752    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"]
Nov 03 15:18:24.328101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.328076    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.372439 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.372402    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"]
Nov 03 15:18:24.376606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.376589    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.389772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.389747    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.389882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.389777    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.389882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.389798    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.389882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.389814    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.389882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.389861    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.390035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.389985    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.390086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.390066    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzdd\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.390121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.390110    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.390217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.390202    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.430709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.430672    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"]
Nov 03 15:18:24.490738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.490694    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.490738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.490742    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.491010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.490767    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.491010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.490800    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.490830    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.491229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491204    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491256    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491331    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491369    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42r96\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491418    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzdd\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.491472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491457    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491509    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491568    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.491838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491642    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491678    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.491838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491719    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.491838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491818    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.492194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.491847    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.492194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.492016    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.492329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.492297    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.492879 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.492854    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.493153 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.493132    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.493378 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.493352    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.493728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.493711    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.495451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.495427    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.510256 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.510222    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"]
Nov 03 15:18:24.553362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.553330    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.554492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.554472    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzdd\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd\") pod \"maas-default-gateway-openshift-default-596dc9545f-n7kp7\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.592404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592318    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592355    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592489    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592532    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592647    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592708    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592735    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592745    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42r96\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592797    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592803    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.592903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.592840    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.593290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.593075    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.593413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.593392    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.593499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.593466    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.594813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.594789    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.595144 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.595127    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.642635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.642603    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:24.656141 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.656114    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42r96\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.656267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.656251    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-kjlhh\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:24.691055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:24.691024    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:25.123728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.123701    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"]
Nov 03 15:18:25.126395 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:18:25.126367    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5a895e_aa24_40ae_94c4_98c7d4f5c64b.slice/crio-3eb4420fba6144af905df37cc02adf5a0f675a153e2b505ac0342aa5a1bad52f WatchSource:0}: Error finding container 3eb4420fba6144af905df37cc02adf5a0f675a153e2b505ac0342aa5a1bad52f: Status 404 returned error can't find the container with id 3eb4420fba6144af905df37cc02adf5a0f675a153e2b505ac0342aa5a1bad52f
Nov 03 15:18:25.128447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.128416    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:18:25.128568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.128484    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:18:25.128568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.128524    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:18:25.175296 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.175272    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"]
Nov 03 15:18:25.175662 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:18:25.175642    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9855361_2a38_4b82_a071_0b9573628dbd.slice/crio-51924fe52187127bf0563e1f2e92fbb18657193fbbcd2a4cd9d4df93dca393dd WatchSource:0}: Error finding container 51924fe52187127bf0563e1f2e92fbb18657193fbbcd2a4cd9d4df93dca393dd: Status 404 returned error can't find the container with id 51924fe52187127bf0563e1f2e92fbb18657193fbbcd2a4cd9d4df93dca393dd
Nov 03 15:18:25.177711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.177681    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:18:25.177801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.177748    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:18:25.177801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.177778    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:18:25.784152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.784105    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" event={"ID":"c9855361-2a38-4b82-a071-0b9573628dbd","Type":"ContainerStarted","Data":"0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83"}
Nov 03 15:18:25.784790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.784764    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" event={"ID":"c9855361-2a38-4b82-a071-0b9573628dbd","Type":"ContainerStarted","Data":"51924fe52187127bf0563e1f2e92fbb18657193fbbcd2a4cd9d4df93dca393dd"}
Nov 03 15:18:25.788220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.788185    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" event={"ID":"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b","Type":"ContainerStarted","Data":"7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190"}
Nov 03 15:18:25.788335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.788243    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" event={"ID":"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b","Type":"ContainerStarted","Data":"3eb4420fba6144af905df37cc02adf5a0f675a153e2b505ac0342aa5a1bad52f"}
Nov 03 15:18:25.977994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:25.963332    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podStartSLOduration=1.963310849 podStartE2EDuration="1.963310849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:25.910947886 +0000 UTC m=+620.835765792" watchObservedRunningTime="2026-11-03 15:18:25.963310849 +0000 UTC m=+620.888128762"
Nov 03 15:18:26.658994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:26.658179    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:26.658994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:26.658368    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:26.658994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:26.658415    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:26.692109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:26.692069    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:26.693256 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:26.693230    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:26.693391 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:26.693285    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:27.643560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:27.643524    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:27.644195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:27.644163    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:27.691425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:27.691397    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:27.691672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:27.691649    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:27.997216 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:18:27.997114    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:18:28.654995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:28.651760    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:28.654995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:28.651841    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:28.691984 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:28.691935    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:28.692153 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:28.692002    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:29.649008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:29.646439    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:29.649008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:29.646531    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:29.692254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:29.692219    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:29.692631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:29.692265    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:30.643639 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:30.643604    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:30.643922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:30.643895    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:30.691537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:30.691498    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:30.691727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:30.691561    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:31.643523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:31.643481    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:31.644078 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:31.643563    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:31.691902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:31.691862    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:31.692137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:31.691934    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:32.643338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:32.643295    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 15:18:32.643548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:32.643367    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 15:18:32.691570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:32.691534    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-kjlhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 03 15:18:32.691776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:32.691597    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 03 15:18:33.832308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:33.832277    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:33.832922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:33.832901    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:33.834255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:33.834235    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:18:34.057139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:34.057088    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podStartSLOduration=10.057075699 podStartE2EDuration="10.057075699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:18:26.053239914 +0000 UTC m=+620.978057828" watchObservedRunningTime="2026-11-03 15:18:34.057075699 +0000 UTC m=+628.981893611"
Nov 03 15:18:34.684157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:34.684121    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:18:34.684409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:34.684378    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:18:35.725152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:35.725112    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-n7kp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:18:35.725555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:35.725164    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:18:36.043033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:36.042924    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:36.043461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:36.043438    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:36.061505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:18:36.061480    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:18:57.938539 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:18:57.938505    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:19:25.724905 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.724862    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"]
Nov 03 15:19:25.728660 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.728635    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.838631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.838592    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-34e8c5-openshift-default-dockercfg-f8ldx\""
Nov 03 15:19:25.841386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.841358    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"]
Nov 03 15:19:25.913364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913326    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znj5v\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913372    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913394    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913571    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913607    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913637    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913753    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913779    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:25.913850 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:25.913812    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.014955    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015022    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015049    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015082    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znj5v\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015106    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015130    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015226    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015256    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015295    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015752    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015827    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015840    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.015998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.015909    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.016055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.016005    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.017839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.017816    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.018038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.018021    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.025372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.025348    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"]
Nov 03 15:19:26.029255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.029236    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.082739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.082675    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.096354 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.096285    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znj5v\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.134120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.134080    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"]
Nov 03 15:19:26.178961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.178913    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bc1f07-openshift-default-dockercfg-lrpxt\""
Nov 03 15:19:26.217586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.217549    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.217797 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.217640    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.217797 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.217707    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.217797 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.217726    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9lj\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.217991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.217894    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.217991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.217943    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.218098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.218046    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.218098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.218081    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.218206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.218117    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319387    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319430    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9lj\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319472    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319495    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319542    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319568    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319598    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319678    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.319766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.319718    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.320105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.320028    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.320105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.320051    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.320299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.320277    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.320392 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.320295    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.320899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.320874    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.322343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.322319    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.322510 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.322493    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.341590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.341553    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:26.356803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.356777    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"]
Nov 03 15:19:26.360631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.360607    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.409594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.406521    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.409594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.409474    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9lj\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.439993    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48tf\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440044    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440059    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440088    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440115    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440132    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440148    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440175    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.442682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.440206    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.454137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.454108    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9855ff-openshift-default-dockercfg-5hjbk\""
Nov 03 15:19:26.493745 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.493703    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"]
Nov 03 15:19:26.541996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.541775    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.546953    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547095    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547181    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48tf\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547253    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547278    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547327    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547377    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547403    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.547434    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.549615    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.549689    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.549882    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.550290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.550101    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.551121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.550818    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.551471 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.551449    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.614839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.614758    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.616941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.616897    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48tf\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.644451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.644419    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:26.674042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.674010    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:26.787653 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.787611    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"]
Nov 03 15:19:26.854055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.854002    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:26.854299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.854130    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:26.854299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:26.854201    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.031100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.031069    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"]
Nov 03 15:19:27.071941 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:19:27.071902    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa52f94_f41d_46fa_8b35_6b72cdbc05c1.slice/crio-78350a3bd1984df38f8861fa6ce5316aad545512941362b987f756efcf79ae38 WatchSource:0}: Error finding container 78350a3bd1984df38f8861fa6ce5316aad545512941362b987f756efcf79ae38: Status 404 returned error can't find the container with id 78350a3bd1984df38f8861fa6ce5316aad545512941362b987f756efcf79ae38
Nov 03 15:19:27.074781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.074739    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.074877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.074815    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.074877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.074846    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.081360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.081331    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" event={"ID":"994acb2b-546b-4ae3-bb39-bd580024257f","Type":"ContainerStarted","Data":"633edd754680d0f86d75f0f8a6f1391ba16fb0e0f3a37afeaa6c501110888a32"}
Nov 03 15:19:27.081495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.081381    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" event={"ID":"994acb2b-546b-4ae3-bb39-bd580024257f","Type":"ContainerStarted","Data":"c58c770b6e264e5bcb26797d0bb905106a6598ef702bdced627916543c8503d2"}
Nov 03 15:19:27.084543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.084517    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"]
Nov 03 15:19:27.118318 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:19:27.118274    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320e09e5_9f50_4f2c_8d41_e2736054eb7f.slice/crio-151752da1ddc020e945184c4633ed95c3fcfd4c32cdf30ad198b9005851d719e WatchSource:0}: Error finding container 151752da1ddc020e945184c4633ed95c3fcfd4c32cdf30ad198b9005851d719e: Status 404 returned error can't find the container with id 151752da1ddc020e945184c4633ed95c3fcfd4c32cdf30ad198b9005851d719e
Nov 03 15:19:27.120929 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.120889    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.121075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.120984    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.121075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.121022    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:27.255753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.255699    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podStartSLOduration=2.255682703 podStartE2EDuration="2.255682703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:27.209419568 +0000 UTC m=+682.134237482" watchObservedRunningTime="2026-11-03 15:19:27.255682703 +0000 UTC m=+682.180500616"
Nov 03 15:19:27.341905 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.341873    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:27.347512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.347483    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:27.365063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.365026    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"]
Nov 03 15:19:27.368752 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.368730    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.427031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.426995    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b841a-openshift-default-dockercfg-bszcv\""
Nov 03 15:19:27.457434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457398    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457524    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5qw\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457570    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457598    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457784    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457862 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457840    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457905 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457889    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.457944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457916    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.458009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.457955    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.462926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.462903    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"]
Nov 03 15:19:27.559780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559746    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.559780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559784    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559809    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559849    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559884    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5qw\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559903    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.559918    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560113    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560180    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560318    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560496    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560552    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560651    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.560795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.560705    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.562512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.562487    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.562956 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.562936    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.625609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.625574    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.625790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.625661    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5qw\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.682437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.682352    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:27.707190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.707161    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"]
Nov 03 15:19:27.710876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.710854    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762320    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762372    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762435    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9mc\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762470    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762555    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762612    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762681    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762715    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.762923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.762743    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.775274 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.775192    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"]
Nov 03 15:19:27.821249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.821219    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f59d26-openshift-default-dockercfg-nkdw5\""
Nov 03 15:19:27.864133 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864102    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864133 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864142    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864168    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864201    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864226    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864282    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9mc\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864314    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864392    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864451    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864593    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864650    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864901 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864721    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864901 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864847    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.864994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.864956    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.867039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.867013    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.867212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.867050    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.930251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.930187    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.938810 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:27.938636    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9mc\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:27.947860 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:19:27.947797    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:19:28.024517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.024486    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:28.091178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.091139    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" event={"ID":"320e09e5-9f50-4f2c-8d41-e2736054eb7f","Type":"ContainerStarted","Data":"01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"}
Nov 03 15:19:28.091316 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.091209    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" event={"ID":"320e09e5-9f50-4f2c-8d41-e2736054eb7f","Type":"ContainerStarted","Data":"151752da1ddc020e945184c4633ed95c3fcfd4c32cdf30ad198b9005851d719e"}
Nov 03 15:19:28.095204 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.095175    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" event={"ID":"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1","Type":"ContainerStarted","Data":"868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"}
Nov 03 15:19:28.095321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.095230    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:28.095321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.095254    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" event={"ID":"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1","Type":"ContainerStarted","Data":"78350a3bd1984df38f8861fa6ce5316aad545512941362b987f756efcf79ae38"}
Nov 03 15:19:28.097533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.096218    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:19:28.097533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.096665    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"]
Nov 03 15:19:28.099792 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:19:28.099770    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c86c97_2e4f_468a_9a41_4f1c73a103a6.slice/crio-8d5a22e211fa1af40820840bfdfee6c5c9f420035e620712c16ded6948391328 WatchSource:0}: Error finding container 8d5a22e211fa1af40820840bfdfee6c5c9f420035e620712c16ded6948391328: Status 404 returned error can't find the container with id 8d5a22e211fa1af40820840bfdfee6c5c9f420035e620712c16ded6948391328
Nov 03 15:19:28.101817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.101788    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:28.101911 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.101853    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:28.101911 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.101890    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:28.224335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.224233    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podStartSLOduration=2.224213026 podStartE2EDuration="2.224213026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:28.222448098 +0000 UTC m=+683.147266011" watchObservedRunningTime="2026-11-03 15:19:28.224213026 +0000 UTC m=+683.149030940"
Nov 03 15:19:28.351367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.351301    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podStartSLOduration=3.351287819 podStartE2EDuration="3.351287819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:28.34975442 +0000 UTC m=+683.274572335" watchObservedRunningTime="2026-11-03 15:19:28.351287819 +0000 UTC m=+683.276105813"
Nov 03 15:19:28.407908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.407875    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"]
Nov 03 15:19:28.409858 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:19:28.409832    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5248002c_1350_4d29_8e36_68a0a2c75328.slice/crio-d9c9154f11663d81ddd82733a37a65cc27c316b8eb9335725d4196dbc0706a8f WatchSource:0}: Error finding container d9c9154f11663d81ddd82733a37a65cc27c316b8eb9335725d4196dbc0706a8f: Status 404 returned error can't find the container with id d9c9154f11663d81ddd82733a37a65cc27c316b8eb9335725d4196dbc0706a8f
Nov 03 15:19:28.412317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.412283    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:28.412429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.412364    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:28.412429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.412409    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:28.644668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.644635    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:28.649699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.649676    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:28.674497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.674472    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:28.679391 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:28.679369    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:29.101901 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.101864    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" event={"ID":"5248002c-1350-4d29-8e36-68a0a2c75328","Type":"ContainerStarted","Data":"49f4998248b3370a34c1dde9dc739b1b1e458f7de8e7f6dc3aea14a5e86a0a21"}
Nov 03 15:19:29.102416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.101917    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" event={"ID":"5248002c-1350-4d29-8e36-68a0a2c75328","Type":"ContainerStarted","Data":"d9c9154f11663d81ddd82733a37a65cc27c316b8eb9335725d4196dbc0706a8f"}
Nov 03 15:19:29.103679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.103639    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" event={"ID":"08c86c97-2e4f-468a-9a41-4f1c73a103a6","Type":"ContainerStarted","Data":"9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"}
Nov 03 15:19:29.104140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.104106    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:29.104257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.104151    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:29.104257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.104177    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" event={"ID":"08c86c97-2e4f-468a-9a41-4f1c73a103a6","Type":"ContainerStarted","Data":"8d5a22e211fa1af40820840bfdfee6c5c9f420035e620712c16ded6948391328"}
Nov 03 15:19:29.104810 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.104791    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:19:29.104894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.104857    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:29.233553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.233500    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podStartSLOduration=2.233486876 podStartE2EDuration="2.233486876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:29.232787445 +0000 UTC m=+684.157605358" watchObservedRunningTime="2026-11-03 15:19:29.233486876 +0000 UTC m=+684.158304788"
Nov 03 15:19:29.642544 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.642473    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podStartSLOduration=2.642455781 podStartE2EDuration="2.642455781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:29.639510539 +0000 UTC m=+684.564328451" watchObservedRunningTime="2026-11-03 15:19:29.642455781 +0000 UTC m=+684.567273694"
Nov 03 15:19:29.682589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.682553    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:29.688046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:29.688022    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:30.025426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:30.025347    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:30.030861 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:30.030835    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:30.109006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:30.108946    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:30.109433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:30.109018    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:30.109532 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:30.109515    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:19:30.109643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:30.109626    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:19:45.527306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:45.527267    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:19:47.806181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.806128    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"]
Nov 03 15:19:47.809989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.809944    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:47.868350 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.868317    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f78fd6-serving-cert\""
Nov 03 15:19:47.870546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.870516    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"]
Nov 03 15:19:47.917212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.917175    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:47.917394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.917237    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhcc\" (UniqueName: \"kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:47.917394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.917349    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:47.917472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:47.917399    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.018803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.018759    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.019017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.018826    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.019017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.018930    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.019129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.019028    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhcc\" (UniqueName: \"kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.019129 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:19:48.019051    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-serving-cert: secret "maas-api-e2e-worker-w2-f78fd6-serving-cert" not found
Nov 03 15:19:48.019129 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:19:48.019126    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls podName:7d06218e-f342-4a0c-8968-f59b54683360 nodeName:}" failed. No retries permitted until 2026-11-03 15:19:48.519108426 +0000 UTC m=+703.443926336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls") pod "maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" (UID: "7d06218e-f342-4a0c-8968-f59b54683360") : secret "maas-api-e2e-worker-w2-f78fd6-serving-cert" not found
Nov 03 15:19:48.019521 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.019496    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.021570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.021549    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.085716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.085626    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhcc\" (UniqueName: \"kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.525098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.525064    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.561767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.561734    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:48.724235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:48.724194    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:49.125038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:49.125015    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"]
Nov 03 15:19:49.126911 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:19:49.126886    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d06218e_f342_4a0c_8968_f59b54683360.slice/crio-0f733ce824ad0842a313c9f801f08d9a37342688eef9667a2f323adca45b5622 WatchSource:0}: Error finding container 0f733ce824ad0842a313c9f801f08d9a37342688eef9667a2f323adca45b5622: Status 404 returned error can't find the container with id 0f733ce824ad0842a313c9f801f08d9a37342688eef9667a2f323adca45b5622
Nov 03 15:19:49.128383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:49.128364    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:19:49.204577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:49.204542    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" event={"ID":"7d06218e-f342-4a0c-8968-f59b54683360","Type":"ContainerStarted","Data":"0f733ce824ad0842a313c9f801f08d9a37342688eef9667a2f323adca45b5622"}
Nov 03 15:19:51.211754 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:51.211718    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" event={"ID":"7d06218e-f342-4a0c-8968-f59b54683360","Type":"ContainerStarted","Data":"eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547"}
Nov 03 15:19:51.212262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:51.211792    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:51.339168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:51.339112    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" podStartSLOduration=3.340660869 podStartE2EDuration="4.339099599s" totalImagesPullingTime="998.43873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:51.33790259 +0000 UTC m=+706.262720499" watchObservedRunningTime="2026-11-03 15:19:51.339099599 +0000 UTC m=+706.263917512"
Nov 03 15:19:54.617335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:54.617237    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:19:55.253921 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:55.253881    4955 generic.go:356] "Generic (PLEG): container finished" podID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerID="01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e" exitCode=0
Nov 03 15:19:55.254291 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:55.254095    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" event={"ID":"320e09e5-9f50-4f2c-8d41-e2736054eb7f","Type":"ContainerDied","Data":"01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"}
Nov 03 15:19:55.254786 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:55.254717    4955 scope.go:122] "RemoveContainer" containerID="01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"
Nov 03 15:19:55.256269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:55.256236    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:55.256372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:55.256317    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:55.256372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:55.256350    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:19:56.281781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:56.281731    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" event={"ID":"320e09e5-9f50-4f2c-8d41-e2736054eb7f","Type":"ContainerStarted","Data":"6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7"}
Nov 03 15:19:56.675358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:56.675322    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:56.675777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:56.675751    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 03 15:19:56.675989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:56.675944    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 03 15:19:57.222694 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:57.222662    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:19:57.678990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:57.678410    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 03 15:19:57.678990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:57.678476    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 03 15:19:57.985318 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:19:57.985221    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:19:58.404356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.404325    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"]
Nov 03 15:19:58.408249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.408230    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.521059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.520427    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtfm\" (UniqueName: \"kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.521059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.520525    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.521059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.520642    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.521059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.520667    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.564190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.564151    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-bc1f07-serving-cert\""
Nov 03 15:19:58.604959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.604925    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"]
Nov 03 15:19:58.621381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.621347    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.621565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.621398    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.621565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.621454    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtfm\" (UniqueName: \"kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.621565 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:19:58.621502    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-serving-cert: secret "maas-api-e2e-worker-w4-bc1f07-serving-cert" not found
Nov 03 15:19:58.621565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.621524    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.621565 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:19:58.621563    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls podName:48814361-2d4f-4e23-b7f0-065c7cae298a nodeName:}" failed. No retries permitted until 2026-11-03 15:19:59.121547988 +0000 UTC m=+714.046365879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls") pod "maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" (UID: "48814361-2d4f-4e23-b7f0-065c7cae298a") : secret "maas-api-e2e-worker-w4-bc1f07-serving-cert" not found
Nov 03 15:19:58.622405 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.622383    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.624678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.624654    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:58.682054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.681328    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 03 15:19:58.682054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.681385    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 03 15:19:58.735433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:58.735011    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtfm\" (UniqueName: \"kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:59.104738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.104644    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:19:59.159209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.158097    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:59.188950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.188543    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:59.321554 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.321509    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:19:59.679999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.679660    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 03 15:19:59.679999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.679742    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 03 15:19:59.768379 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:19:59.761442    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"]
Nov 03 15:19:59.781645 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:19:59.781610    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48814361_2d4f_4e23_b7f0_065c7cae298a.slice/crio-b9b3a9f23a5ef0446683b1904b0846b3b111758fcbf38c154db53b2675df0559 WatchSource:0}: Error finding container b9b3a9f23a5ef0446683b1904b0846b3b111758fcbf38c154db53b2675df0559: Status 404 returned error can't find the container with id b9b3a9f23a5ef0446683b1904b0846b3b111758fcbf38c154db53b2675df0559
Nov 03 15:20:00.304352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:00.304322    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" event={"ID":"48814361-2d4f-4e23-b7f0-065c7cae298a","Type":"ContainerStarted","Data":"b9b3a9f23a5ef0446683b1904b0846b3b111758fcbf38c154db53b2675df0559"}
Nov 03 15:20:00.687221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:00.686701    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 03 15:20:00.687221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:00.686769    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 03 15:20:01.310587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:01.310552    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" event={"ID":"48814361-2d4f-4e23-b7f0-065c7cae298a","Type":"ContainerStarted","Data":"aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034"}
Nov 03 15:20:01.311068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:01.310710    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:20:01.442252 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:01.442196    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" podStartSLOduration=2.541512522 podStartE2EDuration="3.442179287s" totalImagesPullingTime="900.666765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:01.441125695 +0000 UTC m=+716.365943612" watchObservedRunningTime="2026-11-03 15:20:01.442179287 +0000 UTC m=+716.366997200"
Nov 03 15:20:01.767242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:01.767203    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:20:01.769255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:01.769217    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:20:04.326502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:04.326419    4955 generic.go:356] "Generic (PLEG): container finished" podID="994acb2b-546b-4ae3-bb39-bd580024257f" containerID="633edd754680d0f86d75f0f8a6f1391ba16fb0e0f3a37afeaa6c501110888a32" exitCode=0
Nov 03 15:20:04.326502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:04.326466    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" event={"ID":"994acb2b-546b-4ae3-bb39-bd580024257f","Type":"ContainerDied","Data":"633edd754680d0f86d75f0f8a6f1391ba16fb0e0f3a37afeaa6c501110888a32"}
Nov 03 15:20:04.327089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:04.327073    4955 scope.go:122] "RemoveContainer" containerID="633edd754680d0f86d75f0f8a6f1391ba16fb0e0f3a37afeaa6c501110888a32"
Nov 03 15:20:04.328675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:04.328640    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:04.328803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:04.328731    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:04.328803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:04.328779    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:05.210579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:05.210551    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:20:05.358700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:05.358667    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" event={"ID":"994acb2b-546b-4ae3-bb39-bd580024257f","Type":"ContainerStarted","Data":"7ead3a9f2aa693c1fd25fe1c8b5019e0a93c3967c15b8b44eb1ebbab20933806"}
Nov 03 15:20:06.352994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:06.349465    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:20:06.352994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:06.349600    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 15:20:06.352994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:06.349647    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 15:20:07.321540 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:07.321511    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:20:07.364503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:07.364461    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 15:20:07.364783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:07.364754    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 15:20:08.344477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:08.344436    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 15:20:08.345534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:08.345501    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 15:20:09.352997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:09.352374    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 15:20:09.352997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:09.352434    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 15:20:10.345625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:10.345567    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Nov 03 15:20:10.345946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:10.345899    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Nov 03 15:20:11.406355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:11.404313    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:20:11.406355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:11.404917    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:20:11.406355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:11.405895    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:20:11.647029 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:11.646987    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:20:15.154500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.154116    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:20:15.154500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.154191    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:20:15.428646 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.428565    4955 generic.go:356] "Generic (PLEG): container finished" podID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerID="868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639" exitCode=0
Nov 03 15:20:15.428825 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.428643    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" event={"ID":"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1","Type":"ContainerDied","Data":"868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"}
Nov 03 15:20:15.444996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.429149    4955 scope.go:122] "RemoveContainer" containerID="868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"
Nov 03 15:20:15.455815 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.455772    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:15.456072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.455860    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:15.456072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:15.455890    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:16.481380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:16.481350    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" event={"ID":"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1","Type":"ContainerStarted","Data":"39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8"}
Nov 03 15:20:16.665460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:16.665412    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:20:16.665794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:16.665770    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 03 15:20:16.665952 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:16.665929    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 03 15:20:17.649996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:17.647508    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 03 15:20:17.649996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:17.647595    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 03 15:20:18.614687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.614606    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"]
Nov 03 15:20:18.618306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.618286    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.645331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.645299    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 03 15:20:18.645564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.645535    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 03 15:20:18.675868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.675829    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8b841a-serving-cert\""
Nov 03 15:20:18.758440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.758407    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"]
Nov 03 15:20:18.774783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.774749    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.775076 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.775054    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcbg\" (UniqueName: \"kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.775288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.775264    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.775412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.775310    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.876245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.876149    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.876245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.876191    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.876490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.876263    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.876490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.876282    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcbg\" (UniqueName: \"kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.876959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.876932    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.879155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.879132    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.879268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.879182    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:18.958663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:18.958622    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcbg\" (UniqueName: \"kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg\") pod \"maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:19.254233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:19.254187    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:19.655534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:19.655493    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"]
Nov 03 15:20:19.658149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:19.658117    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 03 15:20:19.658287 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:19.658186    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 03 15:20:19.659803 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:20:19.659772    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e6c823_1909_45a1_9fba_ce066d8ef5bb.slice/crio-5317e38cf9f8fd821ef8383e1edb14803728b6ff8c228474de902041fa47c0aa WatchSource:0}: Error finding container 5317e38cf9f8fd821ef8383e1edb14803728b6ff8c228474de902041fa47c0aa: Status 404 returned error can't find the container with id 5317e38cf9f8fd821ef8383e1edb14803728b6ff8c228474de902041fa47c0aa
Nov 03 15:20:20.503800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:20.503771    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx" event={"ID":"71e6c823-1909-45a1-9fba-ce066d8ef5bb","Type":"ContainerStarted","Data":"5317e38cf9f8fd821ef8383e1edb14803728b6ff8c228474de902041fa47c0aa"}
Nov 03 15:20:20.650238 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:20.649830    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 03 15:20:20.650238 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:20.649899    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 03 15:20:21.511505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:21.511471    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx" event={"ID":"71e6c823-1909-45a1-9fba-ce066d8ef5bb","Type":"ContainerStarted","Data":"0a218f1e7b12fbc6379068b5bdf5550e5b101f0ea12be02e4fa27c59e59964d5"}
Nov 03 15:20:21.511928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:21.511535    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:21.640889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:21.640831    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx" podStartSLOduration=2.806361536 podStartE2EDuration="3.640816637s" totalImagesPullingTime="834.455101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:20:21.639380125 +0000 UTC m=+736.564198019" watchObservedRunningTime="2026-11-03 15:20:21.640816637 +0000 UTC m=+736.565634588"
Nov 03 15:20:21.645117 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:21.645079    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 03 15:20:21.645267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:21.645130    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 03 15:20:22.522474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.520823    4955 generic.go:356] "Generic (PLEG): container finished" podID="5248002c-1350-4d29-8e36-68a0a2c75328" containerID="49f4998248b3370a34c1dde9dc739b1b1e458f7de8e7f6dc3aea14a5e86a0a21" exitCode=0
Nov 03 15:20:22.522474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.521581    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" event={"ID":"5248002c-1350-4d29-8e36-68a0a2c75328","Type":"ContainerDied","Data":"49f4998248b3370a34c1dde9dc739b1b1e458f7de8e7f6dc3aea14a5e86a0a21"}
Nov 03 15:20:22.522474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.521886    4955 scope.go:122] "RemoveContainer" containerID="49f4998248b3370a34c1dde9dc739b1b1e458f7de8e7f6dc3aea14a5e86a0a21"
Nov 03 15:20:22.525519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.524689    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:22.525519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.524777    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:22.525519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.524824    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:22.649221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.649193    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:20:22.649504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.649485    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:20:22.650231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:22.650210    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:20:23.528259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:23.528224    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" event={"ID":"5248002c-1350-4d29-8e36-68a0a2c75328","Type":"ContainerStarted","Data":"152564191ceea7eb358f65f5f080c3e366f66065e40529c21d02410b7e00c102"}
Nov 03 15:20:24.025302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:24.025261    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:20:24.025606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:24.025583    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 15:20:24.025716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:24.025641    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 15:20:25.025101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:25.025062    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 15:20:25.025510 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:25.025125    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 15:20:26.025410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:26.025376    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 15:20:26.025816 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:26.025434    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 15:20:26.501247 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:26.501211    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:20:27.025869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:27.025832    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 15:20:27.026407 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:27.025912    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 15:20:27.527338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:27.527302    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:20:27.985261 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:20:27.985194    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:20:28.025333 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:28.025299    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 03 15:20:28.025603 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:28.025363    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 03 15:20:29.037996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:29.033344    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:20:29.037996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:29.034715    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:20:29.056040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:29.046209    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:20:36.318111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.318074    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:20:36.639062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.638951    4955 generic.go:356] "Generic (PLEG): container finished" podID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerID="9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6" exitCode=0
Nov 03 15:20:36.639062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.639027    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" event={"ID":"08c86c97-2e4f-468a-9a41-4f1c73a103a6","Type":"ContainerDied","Data":"9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"}
Nov 03 15:20:36.639431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.639416    4955 scope.go:122] "RemoveContainer" containerID="9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"
Nov 03 15:20:36.640880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.640850    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:36.640938 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.640924    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:36.641003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:36.640951    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:20:37.661258 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:37.661224    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" event={"ID":"08c86c97-2e4f-468a-9a41-4f1c73a103a6","Type":"ContainerStarted","Data":"5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae"}
Nov 03 15:20:37.683137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:37.683102    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:20:37.683297 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:37.683270    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 15:20:37.683339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:37.683321    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 15:20:38.683235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:38.683201    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 15:20:38.683671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:38.683251    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 15:20:39.683322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:39.683284    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 15:20:39.683738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:39.683340    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 15:20:40.683423 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:40.683385    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 15:20:40.683880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:40.683439    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 15:20:41.683058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:41.683027    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 15:20:41.683235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:41.683079    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 15:20:42.682833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:42.682798    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 03 15:20:42.683264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:42.682857    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 03 15:20:43.687581 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:43.687551    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:20:43.688116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:43.687915    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:20:43.688766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:20:43.688740    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:20:57.937834 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:20:57.937795    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:21:02.035518 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:02.035479    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:21:17.316860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:17.316830    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:21:27.939717 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:21:27.939680    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:21:33.431771 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.431738    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"]
Nov 03 15:21:33.435572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.435552    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.494758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.494723    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"]
Nov 03 15:21:33.542043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.542005    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"openshift-service-ca.crt\""
Nov 03 15:21:33.542294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.542072    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"kube-root-ca.crt\""
Nov 03 15:21:33.542294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.542097    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"premium-w1-9855ff-kserve-self-signed-certs\""
Nov 03 15:21:33.542294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.542148    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"default-dockercfg-9n95r\""
Nov 03 15:21:33.568698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.568657    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcc9k\" (UniqueName: \"kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.568866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.568706    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.568866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.568769    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.568866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.568798    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.569014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.568941    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.569052 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.569032    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.669782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.669743    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.669960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.669867    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.669960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.669914    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.670065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.670022    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcc9k\" (UniqueName: \"kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.670111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.670062    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.670111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.670085    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.670348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.670319    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.670433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.670391    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.670584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.670542    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.672214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.672195    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.672644 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.672623    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.733727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.733641    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcc9k\" (UniqueName: \"kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k\") pod \"premium-w1-9855ff-kserve-78f5bdd44d-mh695\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") " pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:33.748060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:33.748019    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:34.128313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:34.128230    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"]
Nov 03 15:21:34.132114 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:21:34.132086    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59910387_1111_4c0b_a838_247b1c8173b3.slice/crio-195f0db547b5f51a2f3cd6e39c4c0b09695d200ea59b3c3d19edb3e5022a6c0b WatchSource:0}: Error finding container 195f0db547b5f51a2f3cd6e39c4c0b09695d200ea59b3c3d19edb3e5022a6c0b: Status 404 returned error can't find the container with id 195f0db547b5f51a2f3cd6e39c4c0b09695d200ea59b3c3d19edb3e5022a6c0b
Nov 03 15:21:34.950254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:34.950222    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" event={"ID":"59910387-1111-4c0b-a838-247b1c8173b3","Type":"ContainerStarted","Data":"7c6139d0d5969877638bee9eb0138033eaaac059bcc5bc2ec1bc9eaf4c11debc"}
Nov 03 15:21:34.950254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:34.950262    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" event={"ID":"59910387-1111-4c0b-a838-247b1c8173b3","Type":"ContainerStarted","Data":"195f0db547b5f51a2f3cd6e39c4c0b09695d200ea59b3c3d19edb3e5022a6c0b"}
Nov 03 15:21:35.069989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:35.069913    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podStartSLOduration=2.069899213 podStartE2EDuration="2.069899213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:21:35.06852146 +0000 UTC m=+809.993339375" watchObservedRunningTime="2026-11-03 15:21:35.069899213 +0000 UTC m=+809.994717126"
Nov 03 15:21:35.221330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:35.221246    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:21:35.752471 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:35.752123    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:43.748490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:43.748457    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:43.761104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:43.761074    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:44.012983 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:44.012886    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:21:49.291658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.291616    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"]
Nov 03 15:21:49.295349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.295326    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.349705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.349673    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.349705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.349710    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.349902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.349732    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.349902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.349838    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.349902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.349876    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9rh\" (UniqueName: \"kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.350106 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.350087    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.354275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.354250    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"]
Nov 03 15:21:49.401817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.401783    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"openshift-service-ca.crt\""
Nov 03 15:21:49.401817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.401787    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"default-dockercfg-sz8vc\""
Nov 03 15:21:49.401817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.401786    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"simulator-w3-8b841a-kserve-self-signed-certs\""
Nov 03 15:21:49.402125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.401856    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"kube-root-ca.crt\""
Nov 03 15:21:49.454993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.454477    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.455184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.455071    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.455184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.455120    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.455761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.455414    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.455761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.455446    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.455761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.455476    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9rh\" (UniqueName: \"kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.455761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.455532    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.456593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.456549    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.457138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.457119    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.462569 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.462513    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.466056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.466030    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.518448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.518413    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9rh\" (UniqueName: \"kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh\") pod \"simulator-w3-8b841a-kserve-d9db7d79d-sv9qr\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") " pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.607772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.607693    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:49.985538 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:49.985499    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"]
Nov 03 15:21:49.989132 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:21:49.989101    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode33ccc80_ef0b_413c_bef5_8b605a025ecc.slice/crio-ba8dd715d46d1f79fe0036ee3d5099b7bf004117cf535003a119d1bb6479298c WatchSource:0}: Error finding container ba8dd715d46d1f79fe0036ee3d5099b7bf004117cf535003a119d1bb6479298c: Status 404 returned error can't find the container with id ba8dd715d46d1f79fe0036ee3d5099b7bf004117cf535003a119d1bb6479298c
Nov 03 15:21:50.038123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:50.038092    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" event={"ID":"e33ccc80-ef0b-413c-bef5-8b605a025ecc","Type":"ContainerStarted","Data":"ba8dd715d46d1f79fe0036ee3d5099b7bf004117cf535003a119d1bb6479298c"}
Nov 03 15:21:51.045453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:51.045407    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" event={"ID":"e33ccc80-ef0b-413c-bef5-8b605a025ecc","Type":"ContainerStarted","Data":"34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979"}
Nov 03 15:21:51.164917 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:51.164845    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podStartSLOduration=2.164831857 podStartE2EDuration="2.164831857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:21:51.163497056 +0000 UTC m=+826.088314970" watchObservedRunningTime="2026-11-03 15:21:51.164831857 +0000 UTC m=+826.089649763"
Nov 03 15:21:51.608616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:51.608575    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:57.952901 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:21:57.952856    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:21:59.608424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:59.608391    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:21:59.620765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:21:59.620737    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:22:00.100643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:00.100616    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:22:01.594602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:01.594559    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:22:19.783803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:19.783767    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:22:27.938698 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:22:27.938658    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:22:43.752411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:43.752374    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:22:49.907423 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:49.907387    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"]
Nov 03 15:22:49.911088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:49.911070    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.014686 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.014660    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"kube-root-ca.crt\""
Nov 03 15:22:50.014867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.014717    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"premium-w4-bc1f07-kserve-self-signed-certs\""
Nov 03 15:22:50.014867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.014689    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"openshift-service-ca.crt\""
Nov 03 15:22:50.014867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.014765    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bc1f07\"/\"default-dockercfg-44z2w\""
Nov 03 15:22:50.016867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.016850    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"]
Nov 03 15:22:50.072584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.072543    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6lb\" (UniqueName: \"kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.072909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.072889    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.073119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.073098    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.073343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.073319    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.073613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.073572    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.073745 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.073630    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.174590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.174496    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.174590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.174534    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.174590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.174559    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6lb\" (UniqueName: \"kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.174590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.174590    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.174941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.174633    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.174941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.174689    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.175074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.175053    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.175114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.175087    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.175150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.175098    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.177050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.177028    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.177323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.177305    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.236385 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.236351    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6lb\" (UniqueName: \"kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb\") pod \"premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") " pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.522192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.522105    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:50.899267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:50.899231    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"]
Nov 03 15:22:50.902952 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:22:50.902922    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda617d72f_fd5b_4830_a2d1_97d589493bb7.slice/crio-f8b557ebd7d342235692f72427e580ce30f1d26ad0d210a9aad8f40c3965b4e3 WatchSource:0}: Error finding container f8b557ebd7d342235692f72427e580ce30f1d26ad0d210a9aad8f40c3965b4e3: Status 404 returned error can't find the container with id f8b557ebd7d342235692f72427e580ce30f1d26ad0d210a9aad8f40c3965b4e3
Nov 03 15:22:51.350632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:51.350538    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" event={"ID":"a617d72f-fd5b-4830-a2d1-97d589493bb7","Type":"ContainerStarted","Data":"a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f"}
Nov 03 15:22:51.350632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:51.350599    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" event={"ID":"a617d72f-fd5b-4830-a2d1-97d589493bb7","Type":"ContainerStarted","Data":"f8b557ebd7d342235692f72427e580ce30f1d26ad0d210a9aad8f40c3965b4e3"}
Nov 03 15:22:51.470990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:51.470929    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podStartSLOduration=2.470913584 podStartE2EDuration="2.470913584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:22:51.469142027 +0000 UTC m=+886.393959940" watchObservedRunningTime="2026-11-03 15:22:51.470913584 +0000 UTC m=+886.395731496"
Nov 03 15:22:51.522415 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:22:51.522372    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:22:57.940204 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:22:57.940157    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:23:00.522788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:00.522757    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:23:00.535493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:00.535469    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:23:01.416677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:01.416649    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:23:05.272091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:05.272057    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:23:05.273122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:05.273103    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:23:05.280992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:05.280951    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:23:05.282025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:05.282002    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:23:15.280261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:15.280223    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:23:27.939009 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:23:27.938935    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:23:31.382474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.382439    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"]
Nov 03 15:23:31.386329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.386312    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.437556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.437520    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.437730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.437560    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rbs\" (UniqueName: \"kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.437730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.437596    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.437828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.437749    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.437882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.437865    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.437958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.437922    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.462054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.462013    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"]
Nov 03 15:23:31.490478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.490445    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b841a\"/\"premium-w3-8b841a-kserve-self-signed-certs\""
Nov 03 15:23:31.548305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.541632    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.548305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.541689    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rbs\" (UniqueName: \"kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.548305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.541726    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.548305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.541763    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.548305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.541850    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.548305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.541922    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.548882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.548773    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.549097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.549069    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.560266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.555380    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.560266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.558464    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.560943 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.560914    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.611600 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.611566    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rbs\" (UniqueName: \"kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs\") pod \"premium-w3-8b841a-kserve-645fdc47cf-cw8nr\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") " pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:31.698261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:31.698218    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:32.092847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:32.092755    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"]
Nov 03 15:23:32.095776 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:23:32.095748    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1c1e120_a539_403d_a6fd_851d2ffa4abc.slice/crio-a026d3650c6b325fff33a7329689d6ca0f7986987236100ac15e64dda6cd3bb2 WatchSource:0}: Error finding container a026d3650c6b325fff33a7329689d6ca0f7986987236100ac15e64dda6cd3bb2: Status 404 returned error can't find the container with id a026d3650c6b325fff33a7329689d6ca0f7986987236100ac15e64dda6cd3bb2
Nov 03 15:23:32.574214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:32.574176    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" event={"ID":"f1c1e120-a539-403d-a6fd-851d2ffa4abc","Type":"ContainerStarted","Data":"1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09"}
Nov 03 15:23:32.574669 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:32.574227    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" event={"ID":"f1c1e120-a539-403d-a6fd-851d2ffa4abc","Type":"ContainerStarted","Data":"a026d3650c6b325fff33a7329689d6ca0f7986987236100ac15e64dda6cd3bb2"}
Nov 03 15:23:32.691275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:32.691225    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podStartSLOduration=1.691211955 podStartE2EDuration="1.691211955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:23:32.690886437 +0000 UTC m=+927.615704354" watchObservedRunningTime="2026-11-03 15:23:32.691211955 +0000 UTC m=+927.616029869"
Nov 03 15:23:32.699081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:32.699058    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:41.698826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:41.698785    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:41.712084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:41.712055    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:42.637279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:42.637249    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:23:55.127340 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:55.127256    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:23:56.003888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:56.003843    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"]
Nov 03 15:23:56.014669 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:56.014123    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy" containerID="cri-o://0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83" gracePeriod=30
Nov 03 15:23:56.058021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:56.057030    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"]
Nov 03 15:23:56.058021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:56.057293    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy" containerID="cri-o://7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190" gracePeriod=30
Nov 03 15:23:56.062014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:56.060240    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"]
Nov 03 15:23:56.062014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:23:56.060496    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy" containerID="cri-o://1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6" gracePeriod=30
Nov 03 15:23:57.939910 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:23:57.939876    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:24:01.343316 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.343286    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:24:01.378208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.378179    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:24:01.407260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407225    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.407464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407296    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.407464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407339    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.407464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407383    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.407464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407411    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.407464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407438    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.407755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407604    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.407755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407736    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.407755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.407747    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.408118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408055    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.408733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408704    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.408835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408748    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.408897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408836    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.408897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408864    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408910    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.408946    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409005    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409038    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token\") pod \"c9855361-2a38-4b82-a071-0b9573628dbd\" (UID: \"c9855361-2a38-4b82-a071-0b9573628dbd\") "
Nov 03 15:24:01.409266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409185    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409228    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409252    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.409419 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409271    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409419 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409269    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:01.409419 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409310    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs\") pod \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\" (UID: \"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b\") "
Nov 03 15:24:01.409583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409551    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409813    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409837    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409855    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409869    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9855361-2a38-4b82-a071-0b9573628dbd-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409884    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409897    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409911    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.410094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.409894    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:01.410586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.410289    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:24:01.410586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.410429    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.411028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.410991    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.411425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.411396    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:24:01.411538 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.411455    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.412616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.412588    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.413288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.413264    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:01.413434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.413416    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:01.413526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.413504    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96" pod "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" (UID: "ca5a895e-aa24-40ae-94c4-98c7d4f5c64b"). InnerVolumeSpecName "kube-api-access-42r96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:01.414135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.414107    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd" pod "c9855361-2a38-4b82-a071-0b9573628dbd" (UID: "c9855361-2a38-4b82-a071-0b9573628dbd"). InnerVolumeSpecName "kube-api-access-bkzdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:01.422691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.422668    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:24:01.510957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.510920    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.510997    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511030    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511110    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511130    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511166    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511209    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511279    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.511458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511312    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.511458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511292    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511393    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket\") pod \"e1e12184-fe13-4680-8917-4054a8bf528e\" (UID: \"e1e12184-fe13-4680-8917-4054a8bf528e\") "
Nov 03 15:24:01.511688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511475    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:01.511688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511523    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.511878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511819    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.511878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511837    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9855361-2a38-4b82-a071-0b9573628dbd-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.511878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511859    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9855361-2a38-4b82-a071-0b9573628dbd-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.511878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511875    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzdd\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-kube-api-access-bkzdd\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511890    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511905    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42r96\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-kube-api-access-42r96\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511920    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511934    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9855361-2a38-4b82-a071-0b9573628dbd-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511947    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511961    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.511995    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.512009    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.512023    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.512037    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.512050    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1e12184-fe13-4680-8917-4054a8bf528e-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.512123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.512065    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.513464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.513439    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:24:01.513577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.513526    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "kube-api-access-x2rzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:01.513812 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.513795    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:24:01.513884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.513822    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token" pod "e1e12184-fe13-4680-8917-4054a8bf528e" (UID: "e1e12184-fe13-4680-8917-4054a8bf528e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:01.613563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.613482    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.613563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.613512    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rzj\" (UniqueName: \"kubernetes.io/projected/e1e12184-fe13-4680-8917-4054a8bf528e-kube-api-access-x2rzj\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.613563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.613523    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.613563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.613534    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1e12184-fe13-4680-8917-4054a8bf528e-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.613563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.613545    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1e12184-fe13-4680-8917-4054a8bf528e-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:24:01.734469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.734433    4955 generic.go:356] "Generic (PLEG): container finished" podID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerID="7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190" exitCode=0
Nov 03 15:24:01.734650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.734491    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"
Nov 03 15:24:01.734650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.734518    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" event={"ID":"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b","Type":"ContainerDied","Data":"7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190"}
Nov 03 15:24:01.734650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.734576    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh" event={"ID":"ca5a895e-aa24-40ae-94c4-98c7d4f5c64b","Type":"ContainerDied","Data":"3eb4420fba6144af905df37cc02adf5a0f675a153e2b505ac0342aa5a1bad52f"}
Nov 03 15:24:01.734650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.734600    4955 scope.go:122] "RemoveContainer" containerID="7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190"
Nov 03 15:24:01.735878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.735848    4955 generic.go:356] "Generic (PLEG): container finished" podID="e1e12184-fe13-4680-8917-4054a8bf528e" containerID="1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6" exitCode=0
Nov 03 15:24:01.736035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.735904    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"
Nov 03 15:24:01.736035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.735898    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" event={"ID":"e1e12184-fe13-4680-8917-4054a8bf528e","Type":"ContainerDied","Data":"1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6"}
Nov 03 15:24:01.736175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.736038    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k" event={"ID":"e1e12184-fe13-4680-8917-4054a8bf528e","Type":"ContainerDied","Data":"84bf42beeda3be21dd7f279c8e62073d59f503b66a7357e28d08efd0926862db"}
Nov 03 15:24:01.737501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.737468    4955 generic.go:356] "Generic (PLEG): container finished" podID="c9855361-2a38-4b82-a071-0b9573628dbd" containerID="0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83" exitCode=0
Nov 03 15:24:01.737622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.737510    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" event={"ID":"c9855361-2a38-4b82-a071-0b9573628dbd","Type":"ContainerDied","Data":"0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83"}
Nov 03 15:24:01.737622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.737568    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7" event={"ID":"c9855361-2a38-4b82-a071-0b9573628dbd","Type":"ContainerDied","Data":"51924fe52187127bf0563e1f2e92fbb18657193fbbcd2a4cd9d4df93dca393dd"}
Nov 03 15:24:01.737622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.737515    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"
Nov 03 15:24:01.746230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.746209    4955 scope.go:122] "RemoveContainer" containerID="7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190"
Nov 03 15:24:01.746623 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:24:01.746602    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190\": container with ID starting with 7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190 not found: ID does not exist" containerID="7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190"
Nov 03 15:24:01.746708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.746635    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190"} err="failed to get container status \"7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190\": rpc error: code = NotFound desc = could not find container \"7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190\": container with ID starting with 7d2fe22fcb702356e8b5988330d71b5530daf9de1a298c001a8bb758c9bfe190 not found: ID does not exist"
Nov 03 15:24:01.746708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.746654    4955 scope.go:122] "RemoveContainer" containerID="1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6"
Nov 03 15:24:01.760081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.760059    4955 scope.go:122] "RemoveContainer" containerID="1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6"
Nov 03 15:24:01.760472 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:24:01.760446    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6\": container with ID starting with 1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6 not found: ID does not exist" containerID="1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6"
Nov 03 15:24:01.760579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.760479    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6"} err="failed to get container status \"1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6\": rpc error: code = NotFound desc = could not find container \"1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6\": container with ID starting with 1ea8cb0ecdea144545dbbe41a6d3e7a92b302be064092db1a24ba6ec201236d6 not found: ID does not exist"
Nov 03 15:24:01.760579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.760504    4955 scope.go:122] "RemoveContainer" containerID="0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83"
Nov 03 15:24:01.771165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.771141    4955 scope.go:122] "RemoveContainer" containerID="0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83"
Nov 03 15:24:01.771509 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:24:01.771485    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83\": container with ID starting with 0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83 not found: ID does not exist" containerID="0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83"
Nov 03 15:24:01.771611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.771517    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83"} err="failed to get container status \"0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83\": rpc error: code = NotFound desc = could not find container \"0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83\": container with ID starting with 0b2fc8ccfd9ec498cce0c8f0b3a3aa4f48470de40535bae9115ec425717c9f83 not found: ID does not exist"
Nov 03 15:24:01.876380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.876294    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"]
Nov 03 15:24:01.880297 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.880270    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-kjlhh"]
Nov 03 15:24:01.992930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.992368    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"]
Nov 03 15:24:01.994661 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:01.994637    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-j4d8k"]
Nov 03 15:24:02.108698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:02.108663    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"]
Nov 03 15:24:02.114569 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:02.114541    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-n7kp7"]
Nov 03 15:24:03.205152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:03.205116    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" path="/var/lib/kubelet/pods/c9855361-2a38-4b82-a071-0b9573628dbd/volumes"
Nov 03 15:24:03.205567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:03.205553    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" path="/var/lib/kubelet/pods/ca5a895e-aa24-40ae-94c4-98c7d4f5c64b/volumes"
Nov 03 15:24:03.205934 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:03.205923    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" path="/var/lib/kubelet/pods/e1e12184-fe13-4680-8917-4054a8bf528e/volumes"
Nov 03 15:24:27.938599 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:24:27.938561    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:24:29.205083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205044    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"]
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205462    4955 cpu_manager.go:414] "containerMap: removing container" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205474    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205483    4955 cpu_manager.go:414] "containerMap: removing container" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205488    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205496    4955 cpu_manager.go:414] "containerMap: removing container" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205501    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205581    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e12184-fe13-4680-8917-4054a8bf528e" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205590    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5a895e-aa24-40ae-94c4-98c7d4f5c64b" containerName="istio-proxy"
Nov 03 15:24:29.205589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.205595    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9855361-2a38-4b82-a071-0b9573628dbd" containerName="istio-proxy"
Nov 03 15:24:29.208579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.208563    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.263092 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.263062    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e62ff3-openshift-default-dockercfg-w2vlc\""
Nov 03 15:24:29.307196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307159    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307198    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307255    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307368    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307424    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307449    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307510    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrjv\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307639 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307625    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.307758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.307731    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.320206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.320172    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"]
Nov 03 15:24:29.408607 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408575    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.408831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408617    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.408831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408640    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.408831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408701    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.408831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408740    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.408831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408773    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.408831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408820    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrjv\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408856    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.408904    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.409170    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.409304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.409304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409514 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.409493    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.409862 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.409837    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.411672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.411643    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.412191 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.412170    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.472437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.472360    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.472570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.472473    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrjv\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.521140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.521107    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:29.948050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.948027    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"]
Nov 03 15:24:29.949651 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:24:29.949623    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bc3475_4053_49bb_a3c9_7d00fc1e9ac5.slice/crio-772327f732070caa2d54e5aa08e2a464a2f0c670bbded37d8b5fe49a583cd3e7 WatchSource:0}: Error finding container 772327f732070caa2d54e5aa08e2a464a2f0c670bbded37d8b5fe49a583cd3e7: Status 404 returned error can't find the container with id 772327f732070caa2d54e5aa08e2a464a2f0c670bbded37d8b5fe49a583cd3e7
Nov 03 15:24:29.952063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.952030    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:24:29.952138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.952099    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:24:29.952138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:29.952129    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:24:30.906469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:30.906434    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" event={"ID":"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5","Type":"ContainerStarted","Data":"a09c136faf744d2121ad26a03dabce4f2a58afa7482c7e05ec38ac86feb977c4"}
Nov 03 15:24:30.906878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:30.906486    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" event={"ID":"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5","Type":"ContainerStarted","Data":"772327f732070caa2d54e5aa08e2a464a2f0c670bbded37d8b5fe49a583cd3e7"}
Nov 03 15:24:31.080936 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:31.080877    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podStartSLOduration=2.080863233 podStartE2EDuration="2.080863233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:31.078962152 +0000 UTC m=+986.003780061" watchObservedRunningTime="2026-11-03 15:24:31.080863233 +0000 UTC m=+986.005681145"
Nov 03 15:24:31.522221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:31.522184    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:31.524055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:31.524027    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 15:24:31.524174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:31.524084    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 15:24:32.521895 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:32.521859    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 15:24:32.522362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:32.521921    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 15:24:33.522227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:33.522186    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 15:24:33.522640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:33.522253    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 15:24:33.885749 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:33.885664    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:24:34.522358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:34.522322    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 15:24:34.522751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:34.522384    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 15:24:35.522450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:35.522414    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 15:24:35.523437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:35.523345    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 15:24:36.687930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:36.687876    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:36.689118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:36.689093    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:36.689882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:36.689863    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:24:43.584361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.584325    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"]
Nov 03 15:24:43.589210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.589192    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.696325 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696274    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"distinct-w2-f78fd6-kserve-self-signed-certs\""
Nov 03 15:24:43.696325 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696275    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"default-dockercfg-6sggv\""
Nov 03 15:24:43.696579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696501    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"openshift-service-ca.crt\""
Nov 03 15:24:43.696579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696539    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f78fd6\"/\"kube-root-ca.crt\""
Nov 03 15:24:43.696708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696686    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.696756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696734    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.696845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696829    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.696906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.696893    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.697031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.697012    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.697173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.697076    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn49m\" (UniqueName: \"kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.698193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.698173    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"]
Nov 03 15:24:43.797671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.797636    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn49m\" (UniqueName: \"kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.797867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.797698    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.797867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.797719    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.797867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.797753    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.798063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.798031    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.798129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.798080    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.798185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.798164    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.798473 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.798443    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.798589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.798471    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.800728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.800697    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.801166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.800785    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.861935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.861863    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn49m\" (UniqueName: \"kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m\") pod \"distinct-w2-f78fd6-kserve-679cb9965b-5qbzx\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") " pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:43.902518 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:43.902484    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:44.315146 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.315121    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"]
Nov 03 15:24:44.317113 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:24:44.317087    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05297afe_2652_4157_8383_bcb8fc7c2851.slice/crio-ab3f43afe7fbbd7cf918f51d9efbbd2617a3381c063e44af94f430a12126285a WatchSource:0}: Error finding container ab3f43afe7fbbd7cf918f51d9efbbd2617a3381c063e44af94f430a12126285a: Status 404 returned error can't find the container with id ab3f43afe7fbbd7cf918f51d9efbbd2617a3381c063e44af94f430a12126285a
Nov 03 15:24:44.618747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.618659    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"]
Nov 03 15:24:44.622397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.622380    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.675999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.675932    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f78fd6-openshift-default-dockercfg-cd4bc\""
Nov 03 15:24:44.708531 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.708485    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.708698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.708554    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.708794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.708726    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.708794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.708774    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbkx\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.708926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.708910    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.708998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.708953    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.709067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.709048    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.709112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.709077    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.709166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.709145    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.728414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.728371    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"]
Nov 03 15:24:44.810018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.809952    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbkx\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810063    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810089    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810109    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810130    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810155    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810198    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810242    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810352    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810642 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810617    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810648    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810817    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.810937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810816    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.811005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.810948    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.812709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.812688    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.812918 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.812902    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.874170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.874098    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbkx\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.874433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.874410    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.935105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.935068    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:44.994842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.994806    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" event={"ID":"05297afe-2652-4157-8383-bcb8fc7c2851","Type":"ContainerStarted","Data":"2ca1a24980d2fd2d3d861a24ccb3f1c7e8ab7f15594586c9d8c3619319be4f02"}
Nov 03 15:24:44.995006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:44.994860    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" event={"ID":"05297afe-2652-4157-8383-bcb8fc7c2851","Type":"ContainerStarted","Data":"ab3f43afe7fbbd7cf918f51d9efbbd2617a3381c063e44af94f430a12126285a"}
Nov 03 15:24:45.116589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:45.116536    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podStartSLOduration=2.116523804 podStartE2EDuration="2.116523804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:45.115706286 +0000 UTC m=+1000.040524201" watchObservedRunningTime="2026-11-03 15:24:45.116523804 +0000 UTC m=+1000.041341717"
Nov 03 15:24:45.363779 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:45.363753    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"]
Nov 03 15:24:45.365843 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:24:45.365816    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b84ae9d_e1cf_4abe_a2af_83108011e47f.slice/crio-07922cfe1b68930eec1f9bcbb0d659aa1818326dace2968346c5ea13755cc92f WatchSource:0}: Error finding container 07922cfe1b68930eec1f9bcbb0d659aa1818326dace2968346c5ea13755cc92f: Status 404 returned error can't find the container with id 07922cfe1b68930eec1f9bcbb0d659aa1818326dace2968346c5ea13755cc92f
Nov 03 15:24:45.368314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:45.368279    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:24:45.368404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:45.368344    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:24:45.368404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:45.368372    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:24:45.903555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:45.903514    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:46.001851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:46.001813    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" event={"ID":"5b84ae9d-e1cf-4abe-a2af-83108011e47f","Type":"ContainerStarted","Data":"738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658"}
Nov 03 15:24:46.002067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:46.001869    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" event={"ID":"5b84ae9d-e1cf-4abe-a2af-83108011e47f","Type":"ContainerStarted","Data":"07922cfe1b68930eec1f9bcbb0d659aa1818326dace2968346c5ea13755cc92f"}
Nov 03 15:24:46.125175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:46.125118    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podStartSLOduration=2.1251030650000002 podStartE2EDuration="2.125103065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:46.124409178 +0000 UTC m=+1001.049227094" watchObservedRunningTime="2026-11-03 15:24:46.125103065 +0000 UTC m=+1001.049921051"
Nov 03 15:24:46.950422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:46.950383    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:46.955099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:46.955063    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 15:24:46.955287 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:46.955119    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 15:24:47.949483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:47.949445    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 15:24:47.949774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:47.949746    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 15:24:48.944995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:48.937437    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 15:24:48.944995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:48.937502    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 15:24:49.953407 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:49.953370    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 15:24:49.954104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:49.954068    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 15:24:50.942426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:50.942385    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 15:24:50.942717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:50.942683    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 15:24:52.043388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:52.043301    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:52.043957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:52.043937    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:52.044614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:52.044597    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:24:53.903355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:53.903323    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:53.915917 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:53.915894    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:54.064828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:24:54.064792    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:24:57.938031 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:24:57.937959    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:25:11.664879 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.664824    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"]
Nov 03 15:25:11.670113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.670087    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"]
Nov 03 15:25:11.670279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.670226    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.674826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.674807    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.676083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.676058    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"]
Nov 03 15:25:11.679598 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.679580    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.726765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.726739    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n82lg\""
Nov 03 15:25:11.734542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.734515    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"]
Nov 03 15:25:11.746129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746097    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.746280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746143    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.746280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746168    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.746280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746221    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.746457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746301    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.746457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746322    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.746457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746383    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.746457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746403    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.746457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746429    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmh9\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.746457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746455    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.746763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746565    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.746763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746636    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.746763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746692    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.746763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746746    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746770    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746794    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746854    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746885    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746903    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.746928    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747043    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.747080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747068    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.747494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747107    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzdc\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.747494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747133    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.747494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747150    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.747494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747205    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.747494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.747254    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6kd\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.789362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.789329    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"]
Nov 03 15:25:11.792264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.792219    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"]
Nov 03 15:25:11.848378 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848340    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.848561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848399    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.848561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848435    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.848561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848460    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.848561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848484    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.848561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848510    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.848561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848531    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848563    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848600    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848626    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848673    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzdc\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848701    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848728    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848768    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848842    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6kd\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848881    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.848893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848888    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849171    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.848887    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849218    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849237    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849276    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849308    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849334    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849333    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849306    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849358    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849385    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849469    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849494    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849502    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849549    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmh9\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849578    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.849597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849584    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.850486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849627    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.850486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849670    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.850486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.849842    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.850486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.850140    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.850486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.850181    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.850752 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.850550    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.850839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.850798    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.850897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.850878    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.851808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.851777    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.851808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.851799    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.851960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.851929    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.852370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.852352    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.852500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.852482    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.853342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.853321    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.913389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.913351    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.958065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.957988    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.958219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.958201    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzdc\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.959218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.959189    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6kd\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd\") pod \"maas-default-gateway-openshift-default-596dc9545f-njfkt\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:11.959384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.959319    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-s7qpc\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:11.959847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.959822    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmh9\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9\") pod \"maas-default-gateway-openshift-default-596dc9545f-nhqvz\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.986907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.986874    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:11.994042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:11.994016    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:12.000158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.000135    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:12.483757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.483724    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"]
Nov 03 15:25:12.485458 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:25:12.485430    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a8f15d_2ed8_47b0_9919_9506b5a1f0a0.slice/crio-7668b94140bcef11073fdde12ec60e0d353c8477fd0efc749c46841d5016e70b WatchSource:0}: Error finding container 7668b94140bcef11073fdde12ec60e0d353c8477fd0efc749c46841d5016e70b: Status 404 returned error can't find the container with id 7668b94140bcef11073fdde12ec60e0d353c8477fd0efc749c46841d5016e70b
Nov 03 15:25:12.485962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.485885    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"]
Nov 03 15:25:12.486062 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:25:12.485963    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61315aa_8eeb_4073_b7e9_b7759356383c.slice/crio-375dc1793c837335c12a310b4b322f82b1473f2847d09fbe0dec3f238fab9050 WatchSource:0}: Error finding container 375dc1793c837335c12a310b4b322f82b1473f2847d09fbe0dec3f238fab9050: Status 404 returned error can't find the container with id 375dc1793c837335c12a310b4b322f82b1473f2847d09fbe0dec3f238fab9050
Nov 03 15:25:12.488064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.488035    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.488143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.488094    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.488143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.488122    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.488429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.488403    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.488477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.488466    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.488528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.488506    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.540322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.540294    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"]
Nov 03 15:25:12.596028 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:25:12.594060    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478b7585_c756_47b3_bf5f_9aa1790c30a5.slice/crio-30cd64ece9c6a40ff072d9821f8379a81cd6c5766dfb9f0ba4899225d745a033 WatchSource:0}: Error finding container 30cd64ece9c6a40ff072d9821f8379a81cd6c5766dfb9f0ba4899225d745a033: Status 404 returned error can't find the container with id 30cd64ece9c6a40ff072d9821f8379a81cd6c5766dfb9f0ba4899225d745a033
Nov 03 15:25:12.597708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.597673    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.597800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.597753    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:12.597800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:12.597786    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:13.215112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.209437    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" event={"ID":"d61315aa-8eeb-4073-b7e9-b7759356383c","Type":"ContainerStarted","Data":"d143ac404c40e9b6e3d7f4a96f54df0b083f2e3b7377d7cee637a35dd44b9979"}
Nov 03 15:25:13.215112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.209517    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" event={"ID":"d61315aa-8eeb-4073-b7e9-b7759356383c","Type":"ContainerStarted","Data":"375dc1793c837335c12a310b4b322f82b1473f2847d09fbe0dec3f238fab9050"}
Nov 03 15:25:13.215112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.209544    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" event={"ID":"478b7585-c756-47b3-bf5f-9aa1790c30a5","Type":"ContainerStarted","Data":"642b2deb22f534b49631d4e4fdcc5d855243bba56baa35e28ef220091d29a321"}
Nov 03 15:25:13.215112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.209583    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" event={"ID":"478b7585-c756-47b3-bf5f-9aa1790c30a5","Type":"ContainerStarted","Data":"30cd64ece9c6a40ff072d9821f8379a81cd6c5766dfb9f0ba4899225d745a033"}
Nov 03 15:25:13.215112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.209607    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" event={"ID":"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0","Type":"ContainerStarted","Data":"7fa573229d73c8b62df6d0f296a775fae097b8d4e0053ceb56f475b437fd25d7"}
Nov 03 15:25:13.215112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.209649    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" event={"ID":"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0","Type":"ContainerStarted","Data":"7668b94140bcef11073fdde12ec60e0d353c8477fd0efc749c46841d5016e70b"}
Nov 03 15:25:13.325197 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.325137    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podStartSLOduration=2.325122054 podStartE2EDuration="2.325122054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:13.323786518 +0000 UTC m=+1028.248604441" watchObservedRunningTime="2026-11-03 15:25:13.325122054 +0000 UTC m=+1028.249939967"
Nov 03 15:25:13.480057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.476815    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podStartSLOduration=2.476792902 podStartE2EDuration="2.476792902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:13.472593066 +0000 UTC m=+1028.397410988" watchObservedRunningTime="2026-11-03 15:25:13.476792902 +0000 UTC m=+1028.401610817"
Nov 03 15:25:13.996277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.995701    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:13.996277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.995883    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:13.996277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.995930    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:13.998230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.998140    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:13.998498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.998479    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:13.998679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:13.998645    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:14.008023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:14.006577    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:14.008023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:14.007960    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:14.008284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:14.008052    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:14.992995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:14.988547    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:14.992995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:14.988610    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:15.007045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:15.000568    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:15.007045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:15.000632    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:15.007045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:15.001348    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:15.007045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:15.001387    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:15.987996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:15.987420    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:15.987996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:15.987521    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:16.005158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.003440    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:16.005158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.003519    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:16.005158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.005048    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:16.005158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.005088    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:16.988994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.988534    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:16.988994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.988617    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:16.995479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.995449    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:16.995750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:16.995713    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:17.001096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:17.001065    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:17.001396 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:17.001305    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:17.989995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:17.987710    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:17.989995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:17.987790    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:18.003819 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.003611    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:18.004224 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.004097    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:18.007123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.003656    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:18.007314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.007288    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:18.549997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.545690    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podStartSLOduration=7.545671926 podStartE2EDuration="7.545671926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:13.60814285 +0000 UTC m=+1028.532960765" watchObservedRunningTime="2026-11-03 15:25:18.545671926 +0000 UTC m=+1033.470489857"
Nov 03 15:25:18.613997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.611293    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:25:18.987596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.987547    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:18.987826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.987636    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:18.995221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.995179    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:18.995707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:18.995242    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:19.000547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:19.000514    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:19.000712 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:19.000576    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:19.988382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:19.988335    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:19.988603 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:19.988408    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:19.994641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:19.994606    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:19.994816 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:19.994672    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:20.001068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:20.001037    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:20.001538 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:20.001094    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:20.988427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:20.988383    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-nhqvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 15:25:20.988622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:20.988454    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 15:25:20.994865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:20.994837    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-s7qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 15:25:20.994922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:20.994900    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 15:25:21.001368 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:21.001331    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-njfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 15:25:21.001841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:21.001392    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 15:25:22.173421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.173387    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:22.175061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.175028    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:22.175911 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.175884    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:25:22.205739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.205698    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:22.206159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.206135    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:22.207063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.207043    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:25:22.283153 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.283113    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:22.283532 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.283513    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:22.284510 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:22.284490    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:25:26.759874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.759841    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"]
Nov 03 15:25:26.763820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.763804    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804309    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bql7\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804347    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804366    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804480    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804544    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804614    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804705    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804764    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.804918 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.804858    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.851382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.851348    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"]
Nov 03 15:25:26.905537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905485    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905534    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905576    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905609    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905660    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905710    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bql7\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905742    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.905802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905767    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.906130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905805    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.906130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.905942    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.906130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.906036    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.906130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.906101    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.906284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.906126    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.906524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.906495    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.908236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.908216    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.908537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.908518    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.968447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.968409    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:26.968619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:26.968511    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bql7\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7\") pod \"maas-default-gateway-openshift-default-596dc9545f-bmbr4\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:27.078238 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:27.078152    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:27.507303 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:27.507275    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"]
Nov 03 15:25:27.509225 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:25:27.509195    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e4d3f5_5f62_4faf_97a5_0bd128a5dded.slice/crio-468c8d0feb6d694b027aece67c9a3dc6fb5c839e8572b905b2c6154930726378 WatchSource:0}: Error finding container 468c8d0feb6d694b027aece67c9a3dc6fb5c839e8572b905b2c6154930726378: Status 404 returned error can't find the container with id 468c8d0feb6d694b027aece67c9a3dc6fb5c839e8572b905b2c6154930726378
Nov 03 15:25:27.511758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:27.511721    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:27.511832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:27.511793    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:27.511832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:27.511824    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:25:27.941089 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:25:27.941048    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:25:28.360697 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:28.360615    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" event={"ID":"36e4d3f5-5f62-4faf-97a5-0bd128a5dded","Type":"ContainerStarted","Data":"a62aaaf82311f48b3c86b526a65e9801af3454bc8a37507f7d6568d463606d54"}
Nov 03 15:25:28.360697 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:28.360666    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" event={"ID":"36e4d3f5-5f62-4faf-97a5-0bd128a5dded","Type":"ContainerStarted","Data":"468c8d0feb6d694b027aece67c9a3dc6fb5c839e8572b905b2c6154930726378"}
Nov 03 15:25:28.538003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:28.536800    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podStartSLOduration=2.5367828230000002 podStartE2EDuration="2.536782823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:28.53326065 +0000 UTC m=+1043.458078564" watchObservedRunningTime="2026-11-03 15:25:28.536782823 +0000 UTC m=+1043.461600740"
Nov 03 15:25:29.078687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:29.078658    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:29.080136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:29.080114    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:29.080200 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:29.080162    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:30.079249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:30.079197    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:30.079636 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:30.079267    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:31.078666 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:31.078629    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:31.078853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:31.078698    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:31.508184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:31.508150    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:25:32.079992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:32.079409    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:32.079992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:32.079477    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:33.078679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:33.078634    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:33.079332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:33.078689    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:34.079302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:34.079272    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:34.079700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:34.079331    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:35.079403 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:35.079372    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-bmbr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 15:25:35.079789 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:35.079432    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 15:25:36.246458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:36.246424    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:36.249016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:36.246902    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:36.249016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:36.247731    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:25:49.357364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:49.357328    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:25:57.939235 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:25:57.939192    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:25:59.920177 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:59.920142    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"]
Nov 03 15:25:59.924295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:25:59.924261    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.030522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.030490    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"]
Nov 03 15:26:00.079164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079124    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.079575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079546    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.079708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079592    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.079708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079652    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.079826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079777    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqqg\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.079882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079824    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.079882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079875    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.080029 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.079909    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.080111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.080092    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181137    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181178    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181230    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181267    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqqg\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181292    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181349    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181378    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181614    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181703    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.181827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181749    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.182044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.182018    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.182263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.181955    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.182452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.182420    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.182660 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.182605    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.183894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.183867    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.184278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.184261    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.243944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.243906    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.244128 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.244030    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqqg\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.539096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.539019    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:00.908568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.908534    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"]
Nov 03 15:26:00.913504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.913478    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:00.971651 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.971620    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"]
Nov 03 15:26:00.992352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.992315    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:00.992518 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.992427    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:00.992518 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.992475    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:00.992672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.992658    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:00.992709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.992693    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ckv\" (UniqueName: \"kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:00.992767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:00.992755    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.018843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.018808    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"distinct-w1-9855ff-kserve-self-signed-certs\""
Nov 03 15:26:01.024885 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.024855    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"]
Nov 03 15:26:01.026622 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:26:01.026596    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121e1692_9acb_41c7_97b3_44a9db147e46.slice/crio-4515c50423291b93b1dd40e4b5d1f6a01949162308058abd9c2cf7bb6e577256 WatchSource:0}: Error finding container 4515c50423291b93b1dd40e4b5d1f6a01949162308058abd9c2cf7bb6e577256: Status 404 returned error can't find the container with id 4515c50423291b93b1dd40e4b5d1f6a01949162308058abd9c2cf7bb6e577256
Nov 03 15:26:01.029001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.028951    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:26:01.029068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.029035    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:26:01.029109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.029069    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:26:01.096549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.096499    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.096740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.096588    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ckv\" (UniqueName: \"kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.096740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.096656    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.100556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.100523    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.105033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.104165    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.105033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.104236    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.105033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.104295    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.105033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.104693    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.105033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.104917    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.121995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.112785    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.121995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.116640    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.161400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.161367    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ckv\" (UniqueName: \"kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv\") pod \"distinct-w1-9855ff-kserve-fd8457759-zjd8h\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.225690 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.225646    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:01.565835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.565753    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" event={"ID":"121e1692-9acb-41c7-97b3-44a9db147e46","Type":"ContainerStarted","Data":"9671b229e36bfe44ea795156c54ab1825f85366807d6846eea92de16bc3a5f04"}
Nov 03 15:26:01.565835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.565816    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" event={"ID":"121e1692-9acb-41c7-97b3-44a9db147e46","Type":"ContainerStarted","Data":"4515c50423291b93b1dd40e4b5d1f6a01949162308058abd9c2cf7bb6e577256"}
Nov 03 15:26:01.607490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.607463    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"]
Nov 03 15:26:01.609270 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:26:01.609241    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f70b0e_88d8_495b_ba4a_72a99a6a6649.slice/crio-47c08b384e5f1cf2b247fd41f49a1f575c370d64e5c69d43e9759732a376bf45 WatchSource:0}: Error finding container 47c08b384e5f1cf2b247fd41f49a1f575c370d64e5c69d43e9759732a376bf45: Status 404 returned error can't find the container with id 47c08b384e5f1cf2b247fd41f49a1f575c370d64e5c69d43e9759732a376bf45
Nov 03 15:26:01.690033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:01.689963    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podStartSLOduration=2.689947518 podStartE2EDuration="2.689947518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:01.688081859 +0000 UTC m=+1076.612899773" watchObservedRunningTime="2026-11-03 15:26:01.689947518 +0000 UTC m=+1076.614765431"
Nov 03 15:26:02.543323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:02.543293    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:02.548375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:02.548345    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 15:26:02.554143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:02.554100    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 15:26:02.573553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:02.573511    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" event={"ID":"27f70b0e-88d8-495b-ba4a-72a99a6a6649","Type":"ContainerStarted","Data":"f72351ec3d6d4c7237740f50af41ba61b73f1068c0742375e17fffe04456ff3a"}
Nov 03 15:26:02.573729 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:02.573564    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" event={"ID":"27f70b0e-88d8-495b-ba4a-72a99a6a6649","Type":"ContainerStarted","Data":"47c08b384e5f1cf2b247fd41f49a1f575c370d64e5c69d43e9759732a376bf45"}
Nov 03 15:26:02.704182 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:02.704126    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podStartSLOduration=2.704112033 podStartE2EDuration="2.704112033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:02.70333074 +0000 UTC m=+1077.628148654" watchObservedRunningTime="2026-11-03 15:26:02.704112033 +0000 UTC m=+1077.628929945"
Nov 03 15:26:03.226646 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:03.226619    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:03.544057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:03.542388    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 15:26:03.544057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:03.542465    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 15:26:04.549996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:04.549599    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 15:26:04.549996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:04.549659    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 15:26:05.552993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:05.551081    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 15:26:05.552993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:05.551141    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 15:26:06.542995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:06.541068    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 15:26:06.542995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:06.541133    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 15:26:07.540349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:07.540313    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 15:26:07.540735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:07.540379    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 15:26:08.543625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:08.543594    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:08.544038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:08.544003    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:08.544680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:08.544663    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:26:11.226529 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:11.226490    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:11.239383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:11.239355    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:11.636695 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:11.636612    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:26:27.044993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:27.044929    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:26:27.948269 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:26:27.948231    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:26:55.121341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.121258    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tzxfq-bnt9n"]
Nov 03 15:26:55.125094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.125077    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.183464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.183434    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tzxfq-bnt9n"]
Nov 03 15:26:55.230814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.230782    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ltlwc\""
Nov 03 15:26:55.233473 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.233449    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.233656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.233572    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsft\" (UniqueName: \"kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.233656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.233606    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.233656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.233639    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.334633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.334597    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.334817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.334730    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.334817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.334772    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsft\" (UniqueName: \"kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.334817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.334797    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.335150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.335134    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.335458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.335439    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.335505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.335440    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.397444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.397408    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsft\" (UniqueName: \"kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft\") pod \"periodic-gathering-tzxfq-bnt9n\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") " pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.436524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.436490    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:26:55.813051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.813014    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tzxfq-bnt9n"]
Nov 03 15:26:55.884037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:55.884002    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n" event={"ID":"aa14640a-b31e-4339-89b0-39d3173adfa7","Type":"ContainerStarted","Data":"551418ed878ca2f48edec2d4983992ff6cf615e4cf20a4b7ac3f44b7c3068864"}
Nov 03 15:26:56.891790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:56.891749    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n" event={"ID":"aa14640a-b31e-4339-89b0-39d3173adfa7","Type":"ContainerStarted","Data":"b4653e7df207dd7acdf17373bb47c1f0d8737df06dbe6c03c59e33fd4775e1fb"}
Nov 03 15:26:57.010720 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:57.010658    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n" podStartSLOduration=2.01064365 podStartE2EDuration="2.01064365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:57.00861736 +0000 UTC m=+1131.933435274" watchObservedRunningTime="2026-11-03 15:26:57.01064365 +0000 UTC m=+1131.935461563"
Nov 03 15:26:57.939728 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:26:57.939690    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:26:58.038434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:58.038403    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:26:59.708322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.708288    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"]
Nov 03 15:26:59.712505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.712483    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.799913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.799864    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.799923    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800041    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800082    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800247 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800190    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800247 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800222    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800246    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800350 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800322    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst5n\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.800392 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.800379    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.815353 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.815322    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"]
Nov 03 15:26:59.902061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902024    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902080    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst5n\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902152    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902222    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902267    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902298    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902436    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902485    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902597    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902625    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902637    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.902829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902806    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.903004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902954    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.903004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.902997    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.904773 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.904751    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.905038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.905020    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.978004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.977918    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:26:59.978139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:26:59.978066    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst5n\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:27:00.026069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.026039    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:27:00.457048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.457018    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"]
Nov 03 15:27:00.458465 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:27:00.458440    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb83b4e_681e_4545_b608_a6927912848f.slice/crio-75770071f95dc6353d22ef2658f84d26182838cba2a113fe584bcb0aa2f7b9b4 WatchSource:0}: Error finding container 75770071f95dc6353d22ef2658f84d26182838cba2a113fe584bcb0aa2f7b9b4: Status 404 returned error can't find the container with id 75770071f95dc6353d22ef2658f84d26182838cba2a113fe584bcb0aa2f7b9b4
Nov 03 15:27:00.460892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.460857    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:27:00.460984 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.460933    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:27:00.460984 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.460961    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:27:00.916708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.916666    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" event={"ID":"fdb83b4e-681e-4545-b608-a6927912848f","Type":"ContainerStarted","Data":"537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf"}
Nov 03 15:27:00.917200 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:00.916739    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" event={"ID":"fdb83b4e-681e-4545-b608-a6927912848f","Type":"ContainerStarted","Data":"75770071f95dc6353d22ef2658f84d26182838cba2a113fe584bcb0aa2f7b9b4"}
Nov 03 15:27:01.026284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:01.026255    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:27:01.027649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:01.027622    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:01.027787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:01.027692    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:01.058994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:01.058168    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podStartSLOduration=2.058149221 podStartE2EDuration="2.058149221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:01.057455453 +0000 UTC m=+1135.982273367" watchObservedRunningTime="2026-11-03 15:27:01.058149221 +0000 UTC m=+1135.982967136"
Nov 03 15:27:02.027236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:02.027200    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:02.027701 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:02.027259    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:03.027088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:03.027057    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:03.027278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:03.027127    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:04.026523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:04.026486    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:04.026710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:04.026537    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:05.026452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:05.026421    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:05.026848 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:05.026466    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:06.027076 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:06.027033    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:06.027503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:06.027105    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:07.026758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:07.026718    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 03 15:27:07.027010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:07.026778    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 03 15:27:08.030885 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:08.030856    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:27:08.031306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:08.031168    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:27:08.031829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:08.031812    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:27:27.939423 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:27:27.939387    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:27:39.407880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.407844    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:27:39.669207 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.669128    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"]
Nov 03 15:27:39.674673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.674654    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.778406 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.778371    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.778406 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.778417    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.778627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.778444    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.778627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.778590    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.778698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.778625    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.778698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.778684    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779v6\" (UniqueName: \"kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.779230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.779208    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"kube-root-ca.crt\""
Nov 03 15:27:39.779230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.779213    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"distinct-2-w0-e62ff3-kserve-self-signed-certs\""
Nov 03 15:27:39.779422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.779216    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"openshift-service-ca.crt\""
Nov 03 15:27:39.779422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.779269    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e62ff3\"/\"default-dockercfg-gw6b2\""
Nov 03 15:27:39.781772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.781754    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"]
Nov 03 15:27:39.886787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.886741    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779v6\" (UniqueName: \"kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.887080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.886954    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.887080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.887027    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.887080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.887068    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.887270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.887135    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.887270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.887174    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.887665 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.887640    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.888430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.888407    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.893272 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.888963    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.903359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.903292    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.903667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.903639    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.951350 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.951271    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779v6\" (UniqueName: \"kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6\") pod \"distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") " pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:39.987182 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:39.987149    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:40.458084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:40.458051    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"]
Nov 03 15:27:40.459568 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:27:40.459540    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a26c442_d934_4dd7_a620_e547742d2cb1.slice/crio-f9b651706e9270588b2baf29665328808fbaab2197c74d2492d6033b6d170baf WatchSource:0}: Error finding container f9b651706e9270588b2baf29665328808fbaab2197c74d2492d6033b6d170baf: Status 404 returned error can't find the container with id f9b651706e9270588b2baf29665328808fbaab2197c74d2492d6033b6d170baf
Nov 03 15:27:41.206744 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:41.206706    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" event={"ID":"7a26c442-d934-4dd7-a620-e547742d2cb1","Type":"ContainerStarted","Data":"2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715"}
Nov 03 15:27:41.206927 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:41.206755    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" event={"ID":"7a26c442-d934-4dd7-a620-e547742d2cb1","Type":"ContainerStarted","Data":"f9b651706e9270588b2baf29665328808fbaab2197c74d2492d6033b6d170baf"}
Nov 03 15:27:41.327949 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:41.327892    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podStartSLOduration=2.327878013 podStartE2EDuration="2.327878013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:41.327781629 +0000 UTC m=+1176.252599544" watchObservedRunningTime="2026-11-03 15:27:41.327878013 +0000 UTC m=+1176.252695926"
Nov 03 15:27:41.988041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:41.988004    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:49.987440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:49.987402    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:50.000158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:50.000127    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:50.273031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:27:50.272948    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:27:57.939214 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:27:57.939174    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:28:05.310671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.310641    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:28:05.311576 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.311550    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:28:05.319358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.319332    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:28:05.320070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.320036    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:28:05.334751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.334725    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"]
Nov 03 15:28:05.338647 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.338631    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.413628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.413600    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"]
Nov 03 15:28:05.442394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.442359    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9855ff\"/\"distinct-2-w1-9855ff-kserve-self-signed-certs\""
Nov 03 15:28:05.509754 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.509715    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.509915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.509788    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.509915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.509845    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.510067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.509928    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.510104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.510061    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxzm\" (UniqueName: \"kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.510104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.510098    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.611810 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.611715    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.611810 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.611793    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.611823    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.611915    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612179 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.612068    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxzm\" (UniqueName: \"kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612179 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.612123    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.612248    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.612292    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.612341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.612318    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.614409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.614382    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.614711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.614690    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.674882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.674845    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxzm\" (UniqueName: \"kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm\") pod \"distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") " pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:05.952412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:05.952378    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:06.332212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:06.332186    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"]
Nov 03 15:28:06.333599 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:28:06.333574    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a27de22_295a_4b0a_bc01_95c05241a6b0.slice/crio-2a05b0c18d4ff83bda235d337194f19b16619438b7878eed5cd15d9377c41092 WatchSource:0}: Error finding container 2a05b0c18d4ff83bda235d337194f19b16619438b7878eed5cd15d9377c41092: Status 404 returned error can't find the container with id 2a05b0c18d4ff83bda235d337194f19b16619438b7878eed5cd15d9377c41092
Nov 03 15:28:06.355864 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:06.355834    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" event={"ID":"7a27de22-295a-4b0a-bc01-95c05241a6b0","Type":"ContainerStarted","Data":"2a05b0c18d4ff83bda235d337194f19b16619438b7878eed5cd15d9377c41092"}
Nov 03 15:28:07.364212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:07.364175    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" event={"ID":"7a27de22-295a-4b0a-bc01-95c05241a6b0","Type":"ContainerStarted","Data":"a015048f1078b11b579f0442c97a480775308817c7e2bbba49e126c382b663e7"}
Nov 03 15:28:07.484562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:07.484509    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podStartSLOduration=2.484494932 podStartE2EDuration="2.484494932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:07.483825033 +0000 UTC m=+1202.408642946" watchObservedRunningTime="2026-11-03 15:28:07.484494932 +0000 UTC m=+1202.409312876"
Nov 03 15:28:07.953081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:07.953046    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:11.562951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:11.562912    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:28:15.953536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:15.953438    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:15.966084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:15.966053    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:16.014430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.014392    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"]
Nov 03 15:28:16.018538 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.018517    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042415    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042494    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042637    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042666    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042815 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042797    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042828    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.042955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.042936    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.043098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.043082    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kj97\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.043158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.043124    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144562    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kj97\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144605    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144625    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144676    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144742    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144764    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.144833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144817    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144899    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.144953    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.145092    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.145151    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.145202    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.145382    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.145721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.145701    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.147032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.147008    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.147322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.147304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.157871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.157836    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"]
Nov 03 15:28:16.206760 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.206682    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.207001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.206944    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kj97\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.333072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.333025    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:16.454925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.454895    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:28:16.765910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.765825    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"]
Nov 03 15:28:16.767094 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:28:16.767069    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf809bb82_5978_4c18_9faa_1d0eb992e650.slice/crio-a6e97d48d7e5fc1fac9bc6c8a48936a9e74a8fb131d2cd0bfe5fa618788b462d WatchSource:0}: Error finding container a6e97d48d7e5fc1fac9bc6c8a48936a9e74a8fb131d2cd0bfe5fa618788b462d: Status 404 returned error can't find the container with id a6e97d48d7e5fc1fac9bc6c8a48936a9e74a8fb131d2cd0bfe5fa618788b462d
Nov 03 15:28:16.770265 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.770210    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:28:16.770392 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.770316    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:28:16.770392 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:16.770365    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:28:17.448004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:17.447946    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" event={"ID":"f809bb82-5978-4c18-9faa-1d0eb992e650","Type":"ContainerStarted","Data":"5e5496bd0d36e938e9785c7afc840e9af41c6a2ad29ca5256096cb521a4bd00d"}
Nov 03 15:28:17.448409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:17.448037    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" event={"ID":"f809bb82-5978-4c18-9faa-1d0eb992e650","Type":"ContainerStarted","Data":"a6e97d48d7e5fc1fac9bc6c8a48936a9e74a8fb131d2cd0bfe5fa618788b462d"}
Nov 03 15:28:17.625386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:17.625330    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podStartSLOduration=2.625314007 podStartE2EDuration="2.625314007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:17.622482963 +0000 UTC m=+1212.547300887" watchObservedRunningTime="2026-11-03 15:28:17.625314007 +0000 UTC m=+1212.550131980"
Nov 03 15:28:18.352140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:18.352100    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:18.352417 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:18.352390    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 03 15:28:18.352580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:18.352552    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 03 15:28:19.354120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:19.354079    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 03 15:28:19.354627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:19.354136    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 03 15:28:20.348995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:20.348584    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 03 15:28:20.348995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:20.348661    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 03 15:28:21.351995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:21.351445    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 03 15:28:21.351995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:21.351510    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 03 15:28:22.338143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:22.337401    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 03 15:28:22.338143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:22.337496    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 03 15:28:23.334368 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:23.334330    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 03 15:28:23.334898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:23.334396    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 03 15:28:24.345790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:24.345756    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:24.346351 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:24.346193    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:24.347162 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:24.347144    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:28:27.939935 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:28:27.939895    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:28:32.167578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.167538    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"]
Nov 03 15:28:32.171707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.171683    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.262376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.262340    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"]
Nov 03 15:28:32.285050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285005    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285069    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285133    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285263    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285287    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285371    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285407    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285432    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkkl\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.285643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.285502    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.386954    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387029    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387049    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387145    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387171    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387191    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387221    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387246    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkkl\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387282    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387657 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387568    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387657 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387603    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387667    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387684    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.387865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.387801    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.389754 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.389729    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.389905 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.389887    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.449708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.449623    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.449844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.449723    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkkl\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:32.544649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.544617    4955 generic.go:356] "Generic (PLEG): container finished" podID="aa14640a-b31e-4339-89b0-39d3173adfa7" containerID="b4653e7df207dd7acdf17373bb47c1f0d8737df06dbe6c03c59e33fd4775e1fb" exitCode=0
Nov 03 15:28:32.544823 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.544695    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n" event={"ID":"aa14640a-b31e-4339-89b0-39d3173adfa7","Type":"ContainerDied","Data":"b4653e7df207dd7acdf17373bb47c1f0d8737df06dbe6c03c59e33fd4775e1fb"}
Nov 03 15:28:32.572057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:32.571996    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:33.005646 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.005616    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"]
Nov 03 15:28:33.007300 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:28:33.007279    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e662a6_fdd8_4e0b_ab1d_194cd4fc3a3d.slice/crio-0771830762b561f42dabd91776f67a193344f3275e704d0f0b82d2df24c61ee5 WatchSource:0}: Error finding container 0771830762b561f42dabd91776f67a193344f3275e704d0f0b82d2df24c61ee5: Status 404 returned error can't find the container with id 0771830762b561f42dabd91776f67a193344f3275e704d0f0b82d2df24c61ee5
Nov 03 15:28:33.010311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.010275    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:28:33.010389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.010345    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:28:33.010389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.010375    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:28:33.553440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.553398    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" event={"ID":"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d","Type":"ContainerStarted","Data":"e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2"}
Nov 03 15:28:33.553942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.553494    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" event={"ID":"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d","Type":"ContainerStarted","Data":"0771830762b561f42dabd91776f67a193344f3275e704d0f0b82d2df24c61ee5"}
Nov 03 15:28:33.587158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.587107    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:33.683154 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.683089    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podStartSLOduration=1.683071336 podStartE2EDuration="1.683071336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:28:33.679409003 +0000 UTC m=+1228.604226942" watchObservedRunningTime="2026-11-03 15:28:33.683071336 +0000 UTC m=+1228.607889249"
Nov 03 15:28:33.769756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.769730    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:28:33.821099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821009    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle\") pod \"aa14640a-b31e-4339-89b0-39d3173adfa7\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") "
Nov 03 15:28:33.821099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821097    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft\" (UniqueName: \"kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft\") pod \"aa14640a-b31e-4339-89b0-39d3173adfa7\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") "
Nov 03 15:28:33.821292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821156    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle\") pod \"aa14640a-b31e-4339-89b0-39d3173adfa7\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") "
Nov 03 15:28:33.821359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821343    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path\") pod \"aa14640a-b31e-4339-89b0-39d3173adfa7\" (UID: \"aa14640a-b31e-4339-89b0-39d3173adfa7\") "
Nov 03 15:28:33.821506 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821475    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle" pod "aa14640a-b31e-4339-89b0-39d3173adfa7" (UID: "aa14640a-b31e-4339-89b0-39d3173adfa7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:28:33.821640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821588    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle" pod "aa14640a-b31e-4339-89b0-39d3173adfa7" (UID: "aa14640a-b31e-4339-89b0-39d3173adfa7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:28:33.821640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821616    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path" pod "aa14640a-b31e-4339-89b0-39d3173adfa7" (UID: "aa14640a-b31e-4339-89b0-39d3173adfa7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:28:33.821863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821844    4955 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-trusted-ca-bundle\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:28:33.821899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821866    4955 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa14640a-b31e-4339-89b0-39d3173adfa7-service-ca-bundle\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:28:33.821899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.821876    4955 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa14640a-b31e-4339-89b0-39d3173adfa7-archives-path\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:28:33.823547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.823523    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft" pod "aa14640a-b31e-4339-89b0-39d3173adfa7" (UID: "aa14640a-b31e-4339-89b0-39d3173adfa7"). InnerVolumeSpecName "kube-api-access-xwsft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:28:33.923595 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:33.923556    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwsft\" (UniqueName: \"kubernetes.io/projected/aa14640a-b31e-4339-89b0-39d3173adfa7-kube-api-access-xwsft\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:28:34.563267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:34.563236    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n"
Nov 03 15:28:34.563761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:34.563266    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tzxfq-bnt9n" event={"ID":"aa14640a-b31e-4339-89b0-39d3173adfa7","Type":"ContainerDied","Data":"551418ed878ca2f48edec2d4983992ff6cf615e4cf20a4b7ac3f44b7c3068864"}
Nov 03 15:28:34.563761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:34.563327    4955 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551418ed878ca2f48edec2d4983992ff6cf615e4cf20a4b7ac3f44b7c3068864"
Nov 03 15:28:34.574313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:34.574287    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 15:28:34.574454 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:34.574333    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 15:28:35.572918 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:35.572886    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 15:28:35.573398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:35.572943    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 15:28:36.572393 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:36.572347    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 15:28:36.572615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:36.572410    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 15:28:37.572992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:37.572947    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 15:28:37.573425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:37.573013    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 15:28:38.572625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:38.572586    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 15:28:38.572823 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:38.572641    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 15:28:39.572842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:39.572799    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 03 15:28:39.573259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:39.572861    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 03 15:28:40.577582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:40.577551    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:40.578080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:40.577978    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:40.578759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:28:40.578739    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:28:57.939583 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:28:57.939545    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:29:01.654748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.654710    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"]
Nov 03 15:29:01.655294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.655278    4955 cpu_manager.go:414] "containerMap: removing container" podUID="aa14640a-b31e-4339-89b0-39d3173adfa7" containerName="insights-gathering"
Nov 03 15:29:01.655338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.655294    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa14640a-b31e-4339-89b0-39d3173adfa7" containerName="insights-gathering"
Nov 03 15:29:01.655393 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.655384    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa14640a-b31e-4339-89b0-39d3173adfa7" containerName="insights-gathering"
Nov 03 15:29:01.658847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.658826    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.728871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.728836    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"]
Nov 03 15:29:01.734435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734402    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734452    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734500    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734523    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734538    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734553    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734635    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4mk\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734670    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.734893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.734718    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.835964 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.835929    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4mk\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836001    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836082    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836173    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836222    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836281    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836312    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836338    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836363    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836553    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.836750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836726    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.837009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.836984    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.837095 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.837026    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.837172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.837154    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.838470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.838440    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.839219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.839188    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.898580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.898546    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.898728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.898591    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4mk\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.974271 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.974188    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:01.995727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:01.995696    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"]
Nov 03 15:29:02.000902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.000829    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.044671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.044631    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.044801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.044701    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.044801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.044755    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.044801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.044782    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.044985 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.044847    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.044985 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.044899    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.045109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.045050    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.045109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.045098    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.045208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.045174    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvhs\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.065318 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.065264    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"]
Nov 03 15:29:02.146684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.146653    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.146875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.146708    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.146948 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.146885    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.146948 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.146920    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147002    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147130    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147213    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147304    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147342    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147356    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147461    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147468    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvhs\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147654    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.147827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.147808    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.149807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.149788    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.150074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.150055    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.210121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.210089    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.210277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.210160    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvhs\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.317809 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.317724    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:02.409669 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.409636    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"]
Nov 03 15:29:02.414518 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:29:02.414487    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7091eb22_7fc0_4a2b_95a8_3a7199144cd1.slice/crio-1e15e85fc0f8baeaf78bf18247df5bb2048affd839a64d86263a53962b12267c WatchSource:0}: Error finding container 1e15e85fc0f8baeaf78bf18247df5bb2048affd839a64d86263a53962b12267c: Status 404 returned error can't find the container with id 1e15e85fc0f8baeaf78bf18247df5bb2048affd839a64d86263a53962b12267c
Nov 03 15:29:02.417784 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.417732    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:02.417907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.417835    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:02.417907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.417882    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:02.709830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.709799    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"]
Nov 03 15:29:02.712687 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:29:02.712656    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a22d35e_eead_4c08_a1ab_b48b074f2d81.slice/crio-8135610fa1f20dc63b055e8dff5164d4b8ab468a830708138bd03f2776a98678 WatchSource:0}: Error finding container 8135610fa1f20dc63b055e8dff5164d4b8ab468a830708138bd03f2776a98678: Status 404 returned error can't find the container with id 8135610fa1f20dc63b055e8dff5164d4b8ab468a830708138bd03f2776a98678
Nov 03 15:29:02.716366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.716321    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:02.716472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.716402    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:02.716472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.716443    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:02.768169 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.768140    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" event={"ID":"1a22d35e-eead-4c08-a1ab-b48b074f2d81","Type":"ContainerStarted","Data":"8135610fa1f20dc63b055e8dff5164d4b8ab468a830708138bd03f2776a98678"}
Nov 03 15:29:02.770551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.770519    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" event={"ID":"7091eb22-7fc0-4a2b-95a8-3a7199144cd1","Type":"ContainerStarted","Data":"53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150"}
Nov 03 15:29:02.770667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.770581    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" event={"ID":"7091eb22-7fc0-4a2b-95a8-3a7199144cd1","Type":"ContainerStarted","Data":"1e15e85fc0f8baeaf78bf18247df5bb2048affd839a64d86263a53962b12267c"}
Nov 03 15:29:02.906732 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.906666    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podStartSLOduration=1.9066512869999999 podStartE2EDuration="1.906651287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:02.905915649 +0000 UTC m=+1257.830733603" watchObservedRunningTime="2026-11-03 15:29:02.906651287 +0000 UTC m=+1257.831469199"
Nov 03 15:29:02.975388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:02.975298    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:03.785951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:03.785917    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" event={"ID":"1a22d35e-eead-4c08-a1ab-b48b074f2d81","Type":"ContainerStarted","Data":"f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1"}
Nov 03 15:29:03.924638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:03.924583    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podStartSLOduration=2.924570896 podStartE2EDuration="2.924570896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:03.922011159 +0000 UTC m=+1258.846829075" watchObservedRunningTime="2026-11-03 15:29:03.924570896 +0000 UTC m=+1258.849388809"
Nov 03 15:29:03.977925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:03.977885    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 03 15:29:03.978225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:03.978198    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 03 15:29:04.318387 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:04.318348    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:04.319584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:04.319563    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 15:29:04.319691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:04.319609    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 15:29:04.977702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:04.977649    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 03 15:29:04.978371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:04.978321    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 03 15:29:05.321028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:05.320943    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 15:29:05.321167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:05.321017    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 15:29:05.977453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:05.977413    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 03 15:29:05.977798 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:05.977768    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 03 15:29:06.318547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:06.318463    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 15:29:06.318547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:06.318516    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 15:29:06.977430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:06.977393    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 03 15:29:06.977737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:06.977701    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 03 15:29:07.318122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:07.318030    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 15:29:07.318122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:07.318102    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 15:29:07.979541 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:07.979502    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 03 15:29:07.979747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:07.979620    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 03 15:29:08.318998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:08.318901    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 15:29:08.319350 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:08.319020    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 15:29:08.985628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:08.985095    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 03 15:29:08.985628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:08.985215    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 03 15:29:09.318592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:09.318497    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 03 15:29:09.318592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:09.318548    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 03 15:29:10.056606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:10.056561    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:10.058101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:10.058075    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:10.062222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:10.062191    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:29:10.339907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:10.339812    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:10.340282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:10.340261    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:10.341419 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:10.341399    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:29:16.166100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:16.166063    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:29:18.659114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:18.659065    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.066055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.057061    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.110906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.110133    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.321247 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.320068    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.452138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.452076    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.681021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.679934    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.725610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.725570    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.726294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.726262    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:19.726782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:19.726766    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:29:20.065054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.057087    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:20.128016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.091795    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:20.128016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.092286    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:29:20.175053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.145156    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:20.175053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.145656    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:29:20.320800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.320699    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:29:20.341884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.341853    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:29:20.410343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.410296    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:20.758998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.656549    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:20.771062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.731646    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:20.804070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.804026    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:29:20.804352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.731696    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Nov 03 15:29:20.805319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.731714    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:29:20.805772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.805745    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:29:20.830040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.731677    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:29:20.830040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.829454    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:29:20.859243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.858733    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:29:20.877555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.877525    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:29:20.901856 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.901824    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:29:20.932227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:20.932193    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:29:21.036066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:21.001715    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:29:21.072228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:21.072199    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:29:21.140506 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:21.140479    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:29:27.940603 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:29:27.940569    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:29:46.031448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.031362    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"]
Nov 03 15:29:46.036592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.036571    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.070740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.070704    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.070916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.070900    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.070960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.070927    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.070960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.070945    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.071056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.071047    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.071099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.071068    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klxn\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.071134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.071101    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.071259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.071237    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.071321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.071288    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.154904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.154868    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"]
Nov 03 15:29:46.173746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.173708    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.173746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.173759    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.174747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.173785    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.174747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.173984    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.174747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.174018    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klxn\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.174747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.174101    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.174747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.174184    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.175117 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.175066    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.175117 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.175111    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.177741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.176881    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.177741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.177145    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.177741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.177358    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.177741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.177626    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.177741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.177698    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.178123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.177932    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.179136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.179113    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.236569 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.236539    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.239666 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.239635    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klxn\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn\") pod \"e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") " pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.350478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.350397    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:46.742336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.742305    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"]
Nov 03 15:29:46.747410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.747390    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.782511 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.782477    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.782678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.782547    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.782678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.782660    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt22\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.782772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.782698    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.782830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.782816    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.782913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.782882    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.783057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.783038    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.783131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.783091    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.783253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.783216    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.787028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.787005    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"]
Nov 03 15:29:46.793863 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:29:46.793836    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb14684e_4893_48b6_9c94_ffa657288564.slice/crio-1e419a7f7e763a192d2a769c572d84117fa97fab50b204ff4ece030dae2bbba1 WatchSource:0}: Error finding container 1e419a7f7e763a192d2a769c572d84117fa97fab50b204ff4ece030dae2bbba1: Status 404 returned error can't find the container with id 1e419a7f7e763a192d2a769c572d84117fa97fab50b204ff4ece030dae2bbba1
Nov 03 15:29:46.800950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.800911    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:46.801075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.801022    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:46.801075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.801059    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:46.838466 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.838436    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"]
Nov 03 15:29:46.884414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884386    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884450    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884483    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt22\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884511    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884566    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884617    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884691    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884723    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.884756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884742    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.885049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884816    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.885049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.884914    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.885218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.885206    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.885426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.885406    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.885426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.885418    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.887665 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.887638    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.888010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.887963    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.947875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.947839    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:46.948069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:46.947960    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt22\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22\") pod \"e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") " pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:47.061498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.061405    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:47.398875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.398839    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" event={"ID":"cb14684e-4893-48b6-9c94-ffa657288564","Type":"ContainerStarted","Data":"f4f192f2b58ef55472cba93d44b87f5a530a96c4c5eccc4f4b7defaf8eb66494"}
Nov 03 15:29:47.399070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.398901    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" event={"ID":"cb14684e-4893-48b6-9c94-ffa657288564","Type":"ContainerStarted","Data":"1e419a7f7e763a192d2a769c572d84117fa97fab50b204ff4ece030dae2bbba1"}
Nov 03 15:29:47.498163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.498134    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"]
Nov 03 15:29:47.498662 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:29:47.498635    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f98bb0_5fbb_475c_b269_508c01f458f4.slice/crio-9788a1bea5b66c5961cc2b82d680b69e0849d906048ac2722f53d7de8acd3d1b WatchSource:0}: Error finding container 9788a1bea5b66c5961cc2b82d680b69e0849d906048ac2722f53d7de8acd3d1b: Status 404 returned error can't find the container with id 9788a1bea5b66c5961cc2b82d680b69e0849d906048ac2722f53d7de8acd3d1b
Nov 03 15:29:47.501245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.501173    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:47.501401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.501266    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:47.501401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.501310    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:29:47.521669 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:47.521622    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podStartSLOduration=2.521607533 podStartE2EDuration="2.521607533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:47.520016065 +0000 UTC m=+1302.444833978" watchObservedRunningTime="2026-11-03 15:29:47.521607533 +0000 UTC m=+1302.446425446"
Nov 03 15:29:48.352994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:48.352356    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:48.365046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:48.363140    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:48.365046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:48.363226    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:48.464771 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:48.464737    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" event={"ID":"05f98bb0-5fbb-475c-b269-508c01f458f4","Type":"ContainerStarted","Data":"d1726b39da778f8a4739dc74e1aa9bb78116dc0a839e24124313af271df409b0"}
Nov 03 15:29:48.465158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:48.465071    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" event={"ID":"05f98bb0-5fbb-475c-b269-508c01f458f4","Type":"ContainerStarted","Data":"9788a1bea5b66c5961cc2b82d680b69e0849d906048ac2722f53d7de8acd3d1b"}
Nov 03 15:29:48.612347 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:48.612202    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podStartSLOduration=2.612185254 podStartE2EDuration="2.612185254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:29:48.611674009 +0000 UTC m=+1303.536491922" watchObservedRunningTime="2026-11-03 15:29:48.612185254 +0000 UTC m=+1303.537003183"
Nov 03 15:29:49.076994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:49.063383    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:29:49.076994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:49.073224    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:49.076994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:49.073303    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:49.353057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:49.352474    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:49.353057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:49.352551    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:50.064655 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:50.064613    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:50.064993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:50.064927    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:50.358060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:50.354351    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:50.358060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:50.354426    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:51.065931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.065891    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:51.066273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.066240    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:51.357049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.352023    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:51.357049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.352091    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:51.870996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.870488    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:51.909997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.909073    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:51.976995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:51.976110    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.033689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.013540    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.033689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.013931    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.086002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.069429    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:52.086002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.069493    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:52.248075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.247991    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 03 15:29:52.352997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.352938    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:52.353245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.353027    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:52.441063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.441018    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.701186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.701128    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.888208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.888160    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.889204 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.888931    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.890803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.889619    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:29:52.910194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.910121    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.910724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.910691    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:29:52.979051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.977327    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:52.979051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:52.977795    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:29:53.052209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.052162    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:53.053108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.053076    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:53.074038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.065225    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:29:53.074463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.074424    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:29:53.101279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.084349    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:53.139062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.125828    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:53.139062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.125911    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:53.178021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.147102    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:29:53.178021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.147182    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:53.413057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.409120    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:29:53.449998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.430740    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:53.450377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.450336    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:53.458511 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.458470    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:29:53.896060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.890118    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:29:53.913726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.913661    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:29:53.914993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.914935    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:29:53.951055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.945597    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:29:53.984112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:53.984061    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:54.082093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.082056    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:29:54.133658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.108068    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:29:54.134147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.108245    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:29:54.160046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.134902    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:29:54.187000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.142354    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:54.293120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.271824    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:54.414060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.393371    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:54.414060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.393437    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:54.417190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.417142    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:54.433006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.417871    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:29:54.522272 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.522197    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:29:54.542459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.542431    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:29:54.608921 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.608874    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:29:54.609414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.609276    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:54.759174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.759067    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:54.924134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.924082    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:54.949711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.949065    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:29:54.962002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.949566    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:55.014556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.949617    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:55.015037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:54.990142    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:55.068997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.061840    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:55.068997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.061912    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:55.152152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.152106    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:55.152730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.152700    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:29:55.153155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.153131    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:29:55.421165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.421124    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:55.509125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.509063    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:55.509431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.460155    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:29:55.510593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.460679    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:29:55.510593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.510113    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:55.580328 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.554829    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:55.993040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:55.992988    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:29:56.046999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.024353    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:56.047610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.027954    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:29:56.182999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.079310    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:56.299958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.299856    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:56.300273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.160458    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:56.300689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.137008    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:56.301079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.136898    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:29:56.301234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.301209    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:56.301375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.160497    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:29:56.301691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.160516    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:29:56.376049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.370834    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:56.376049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.370894    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:56.392228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.392162    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:29:56.392573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.392526    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:56.521666 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.521615    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:29:56.624058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:56.588483    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:57.016997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.007495    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Nov 03 15:29:57.016997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.009353    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:57.056096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.049525    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:29:57.056096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.050015    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:29:57.109440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.065679    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:57.109440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.065743    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:57.313851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.313763    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:29:57.415496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.415457    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:57.415689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.415520    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:57.422353 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.422198    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:29:57.430047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.430007    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:29:57.596781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.596688    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:29:57.599276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.599255    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:29:57.600018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.600000    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:29:57.600703 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.600687    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:29:57.601376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.601361    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:29:57.615394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.615271    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:29:57.667777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.667748    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:29:57.713880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.713850    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:29:57.720773 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:57.720746    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:29:57.970289 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:29:57.970173    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:29:58.064798 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:58.064766    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:58.065545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:58.065380    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:58.351372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:58.351292    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 15:29:58.351372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:58.351350    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 15:29:59.062362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:59.062323    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 03 15:29:59.062542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:59.062391    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 03 15:29:59.356280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:59.356195    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:59.356696 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:59.356545    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:29:59.357435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:29:59.357416    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:30:00.225016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.224955    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"]
Nov 03 15:30:00.229319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.229281    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.252051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.252016    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"]
Nov 03 15:30:00.256995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.256950    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.282718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.282684    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\""
Nov 03 15:30:00.322155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.322081    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"]
Nov 03 15:30:00.366041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.355855    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfln\" (UniqueName: \"kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln\") pod \"maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.366041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.358603    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.366041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.358666    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895330-hxgvx\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.366041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.358744    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhn8m\" (UniqueName: \"kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m\") pod \"maas-api-key-cleanup-29895330-hxgvx\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.366041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.356761    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"]
Nov 03 15:30:00.460161 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.460125    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.460367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.460173    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895330-hxgvx\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.460367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.460226    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhn8m\" (UniqueName: \"kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m\") pod \"maas-api-key-cleanup-29895330-hxgvx\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.460503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.460436    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfln\" (UniqueName: \"kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln\") pod \"maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.461361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.461003    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895330-hxgvx\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.461361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.461304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.523276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.523169    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfln\" (UniqueName: \"kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln\") pod \"maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.523445 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.523280    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhn8m\" (UniqueName: \"kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m\") pod \"maas-api-key-cleanup-29895330-hxgvx\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:00.547999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.545232    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:00.580442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:00.580405    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:01.045274 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.045244    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"]
Nov 03 15:30:01.047797 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:01.047742    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f91448a_36f7_497d_b013_0efa7997b82d.slice/crio-a94b759df2e231ae1d6fb5b29dfcdbb9f64d86cf9423d21848efcf4fa3d3d3ed WatchSource:0}: Error finding container a94b759df2e231ae1d6fb5b29dfcdbb9f64d86cf9423d21848efcf4fa3d3d3ed: Status 404 returned error can't find the container with id a94b759df2e231ae1d6fb5b29dfcdbb9f64d86cf9423d21848efcf4fa3d3d3ed
Nov 03 15:30:01.050552 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.050531    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:30:01.088142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.088101    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:01.088322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.088180    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:01.089813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.089782    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:30:01.090380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.090350    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:30:01.094578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.094545    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"]
Nov 03 15:30:01.096044 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:01.096021    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44880eaf_aec3_45b9_9b28_3930d6a4bad4.slice/crio-04d23112ec5ecc97ef445b40b6b3d27acfde1fc0cb84b890c823acdd52770c37 WatchSource:0}: Error finding container 04d23112ec5ecc97ef445b40b6b3d27acfde1fc0cb84b890c823acdd52770c37: Status 404 returned error can't find the container with id 04d23112ec5ecc97ef445b40b6b3d27acfde1fc0cb84b890c823acdd52770c37
Nov 03 15:30:01.141892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.141861    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:30:01.412728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.412692    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" event={"ID":"4f91448a-36f7-497d-b013-0efa7997b82d","Type":"ContainerStarted","Data":"a94b759df2e231ae1d6fb5b29dfcdbb9f64d86cf9423d21848efcf4fa3d3d3ed"}
Nov 03 15:30:01.414017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.413995    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerStarted","Data":"04d23112ec5ecc97ef445b40b6b3d27acfde1fc0cb84b890c823acdd52770c37"}
Nov 03 15:30:01.481282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.481249    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"]
Nov 03 15:30:01.486424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.486402    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.548861 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.548823    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"]
Nov 03 15:30:01.601940 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.601905    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.601940 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.601942    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602054    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602097    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602123    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602221    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602259    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602277    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.602508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.602403    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxpl\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704258 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704158    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704258 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704203    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704258 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704224    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704276    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxpl\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704435    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704464    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704512    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704573    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704607    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.704820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.704648    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.705037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.705014    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.705103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.705024    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.705195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.705171    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.705263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.705244    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.707327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.707304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.707440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.707422    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.766205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.766175    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.766347 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.766244    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxpl\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:01.804667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:01.804631    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:02.200759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.200716    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"]
Nov 03 15:30:02.205432 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:02.205401    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f7618_681b_4acf_8651_0805f1de1202.slice/crio-61f3e519433aab739853727d89ac0a91faf41899e55d715b2273d73c12e16d62 WatchSource:0}: Error finding container 61f3e519433aab739853727d89ac0a91faf41899e55d715b2273d73c12e16d62: Status 404 returned error can't find the container with id 61f3e519433aab739853727d89ac0a91faf41899e55d715b2273d73c12e16d62
Nov 03 15:30:02.208024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.207954    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:02.208112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.208079    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:02.208157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.208123    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:02.423156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.423118    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" event={"ID":"508f7618-681b-4acf-8651-0805f1de1202","Type":"ContainerStarted","Data":"19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed"}
Nov 03 15:30:02.423630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.423189    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" event={"ID":"508f7618-681b-4acf-8651-0805f1de1202","Type":"ContainerStarted","Data":"61f3e519433aab739853727d89ac0a91faf41899e55d715b2273d73c12e16d62"}
Nov 03 15:30:02.546459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.546353    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podStartSLOduration=1.546334523 podStartE2EDuration="1.546334523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:02.545393394 +0000 UTC m=+1317.470211311" watchObservedRunningTime="2026-11-03 15:30:02.546334523 +0000 UTC m=+1317.471152435"
Nov 03 15:30:02.805530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:02.805452    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:03.807132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:03.807096    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:30:03.807532 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:03.807159    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:30:04.805285 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:04.805236    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:30:04.805489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:04.805299    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:30:05.805329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:05.805293    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:30:05.805951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:05.805374    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:30:06.805677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:06.805641    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:30:06.806189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:06.805703    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:30:07.805491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:07.805455    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:30:07.805675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:07.805521    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:30:08.806075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:08.806035    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:30:08.806499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:08.806102    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:30:09.856330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:09.856281    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:09.860549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:09.860406    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:09.860549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:09.860491    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:30:10.504864 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:10.504780    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerStarted","Data":"9c50a805075255948426c09d8b83383b8e6d2b9fc89e0c28e2ea1d61cf745027"}
Nov 03 15:30:10.623149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:10.623100    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" podStartSLOduration=1.74021824 podStartE2EDuration="10.623085487s" totalImagesPullingTime="8.882867247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:10.620704311 +0000 UTC m=+1325.545522225" watchObservedRunningTime="2026-11-03 15:30:10.623085487 +0000 UTC m=+1325.547903401"
Nov 03 15:30:11.512381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:11.512345    4955 generic.go:356] "Generic (PLEG): container finished" podID="4f91448a-36f7-497d-b013-0efa7997b82d" containerID="0d604cf8a7ab6a2e4940d09029586e6a053d0ce6f91cee0ba48a86401a264f2b" exitCode=60
Nov 03 15:30:11.512843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:11.512430    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" event={"ID":"4f91448a-36f7-497d-b013-0efa7997b82d","Type":"ContainerDied","Data":"0d604cf8a7ab6a2e4940d09029586e6a053d0ce6f91cee0ba48a86401a264f2b"}
Nov 03 15:30:11.512843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:11.512794    4955 scope.go:122] "RemoveContainer" containerID="0d604cf8a7ab6a2e4940d09029586e6a053d0ce6f91cee0ba48a86401a264f2b"
Nov 03 15:30:11.515030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:11.514424    4955 generic.go:356] "Generic (PLEG): container finished" podID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerID="9c50a805075255948426c09d8b83383b8e6d2b9fc89e0c28e2ea1d61cf745027" exitCode=60
Nov 03 15:30:11.515030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:11.514533    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerDied","Data":"9c50a805075255948426c09d8b83383b8e6d2b9fc89e0c28e2ea1d61cf745027"}
Nov 03 15:30:11.515030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:11.514786    4955 scope.go:122] "RemoveContainer" containerID="9c50a805075255948426c09d8b83383b8e6d2b9fc89e0c28e2ea1d61cf745027"
Nov 03 15:30:12.523750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:12.523712    4955 generic.go:356] "Generic (PLEG): container finished" podID="4f91448a-36f7-497d-b013-0efa7997b82d" containerID="72249dc53c7404c96c89253ed8bda8b24548992d082f90c5c946dae310cef6ba" exitCode=60
Nov 03 15:30:12.524248 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:12.523793    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" event={"ID":"4f91448a-36f7-497d-b013-0efa7997b82d","Type":"ContainerDied","Data":"72249dc53c7404c96c89253ed8bda8b24548992d082f90c5c946dae310cef6ba"}
Nov 03 15:30:12.524248 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:12.523888    4955 scope.go:122] "RemoveContainer" containerID="0d604cf8a7ab6a2e4940d09029586e6a053d0ce6f91cee0ba48a86401a264f2b"
Nov 03 15:30:12.524248 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:12.524243    4955 scope.go:122] "RemoveContainer" containerID="72249dc53c7404c96c89253ed8bda8b24548992d082f90c5c946dae310cef6ba"
Nov 03 15:30:12.524609 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:12.524586    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56_odh-ai-gateway-infra(4f91448a-36f7-497d-b013-0efa7997b82d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" podUID="4f91448a-36f7-497d-b013-0efa7997b82d"
Nov 03 15:30:12.526112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:12.526069    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerStarted","Data":"abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f"}
Nov 03 15:30:13.535987 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:13.535950    4955 scope.go:122] "RemoveContainer" containerID="72249dc53c7404c96c89253ed8bda8b24548992d082f90c5c946dae310cef6ba"
Nov 03 15:30:13.536450 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:13.536356    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56_odh-ai-gateway-infra(4f91448a-36f7-497d-b013-0efa7997b82d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" podUID="4f91448a-36f7-497d-b013-0efa7997b82d"
Nov 03 15:30:13.538439 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:13.538411    4955 generic.go:356] "Generic (PLEG): container finished" podID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerID="abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f" exitCode=60
Nov 03 15:30:13.538572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:13.538486    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerDied","Data":"abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f"}
Nov 03 15:30:13.538572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:13.538548    4955 scope.go:122] "RemoveContainer" containerID="9c50a805075255948426c09d8b83383b8e6d2b9fc89e0c28e2ea1d61cf745027"
Nov 03 15:30:13.538761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:13.538745    4955 scope.go:122] "RemoveContainer" containerID="abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f"
Nov 03 15:30:13.539141 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:13.539109    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895330-hxgvx_odh-ai-gateway-infra(44880eaf-aec3-45b9-9b28-3930d6a4bad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4"
Nov 03 15:30:14.024217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.024184    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"]
Nov 03 15:30:14.024466 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.024437    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy" containerID="cri-o://642b2deb22f534b49631d4e4fdcc5d855243bba56baa35e28ef220091d29a321" gracePeriod=30
Nov 03 15:30:14.028741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.028693    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"]
Nov 03 15:30:14.028976 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.028945    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy" containerID="cri-o://7fa573229d73c8b62df6d0f296a775fae097b8d4e0053ceb56f475b437fd25d7" gracePeriod=30
Nov 03 15:30:14.031766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.031744    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"]
Nov 03 15:30:14.031924 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.031907    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy" containerID="cri-o://d143ac404c40e9b6e3d7f4a96f54df0b083f2e3b7377d7cee637a35dd44b9979" gracePeriod=30
Nov 03 15:30:14.072577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.072544    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"]
Nov 03 15:30:14.072806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.072749    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy" containerID="cri-o://a62aaaf82311f48b3c86b526a65e9801af3454bc8a37507f7d6568d463606d54" gracePeriod=30
Nov 03 15:30:14.546302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:14.546278    4955 scope.go:122] "RemoveContainer" containerID="abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f"
Nov 03 15:30:14.546682 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:14.546572    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895330-hxgvx_odh-ai-gateway-infra(44880eaf-aec3-45b9-9b28-3930d6a4bad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4"
Nov 03 15:30:15.193292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.193249    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"]
Nov 03 15:30:15.203905 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.203871    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.264476 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.264440    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"]
Nov 03 15:30:15.356997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.331020    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-47b989b3-openshift-default-dockercfg-7snsq\""
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.363753    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.363806    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.363835    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj4p\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.363910    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.363990    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.364039    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.364170    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.364345    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.378037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.364478    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465497    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465567    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465619    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465635    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465652    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj4p\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465696    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465741    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465768    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.465837    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.466165    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.466584    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.466733    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.467297    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.467845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.467543    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.469480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.469455    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.475242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.475218    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.595777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.595744    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj4p\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.598356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.598320    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token\") pod \"e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") " pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:15.821547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:15.821471    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:16.110181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.110116    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"]
Nov 03 15:30:16.126316 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.126281    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.190222 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.190185    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"]
Nov 03 15:30:16.309746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.309710    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.309899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.309761    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.309960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.309923    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.310044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.310028    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.310212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.310198    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.310305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.310226    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.310305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.310257    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.310418 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.310320    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mwv\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.310478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.310415    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.411740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.411708    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.411910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.411781    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.411910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.411855    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.411910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.411893    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.411919    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.411948    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mwv\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412046    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412235    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412351    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412388    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412535    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412555    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412624    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.412721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.412694    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.414367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.414345    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.414651 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.414635    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.456741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.456716    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"]
Nov 03 15:30:16.457350 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:16.457327    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6548c2a_1484_4966_a9b9_abb8c70ae652.slice/crio-806b95634ab00bd4bc204529c14100283fe9f81c85249bfb0efb6d1a4c85bacd WatchSource:0}: Error finding container 806b95634ab00bd4bc204529c14100283fe9f81c85249bfb0efb6d1a4c85bacd: Status 404 returned error can't find the container with id 806b95634ab00bd4bc204529c14100283fe9f81c85249bfb0efb6d1a4c85bacd
Nov 03 15:30:16.459832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.459797    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:16.459927 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.459881    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:16.460022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.459928    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:16.475553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.475527    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.476209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.476185    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mwv\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.597924 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.597884    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" event={"ID":"d6548c2a-1484-4966-a9b9-abb8c70ae652","Type":"ContainerStarted","Data":"f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"}
Nov 03 15:30:16.598463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.597947    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" event={"ID":"d6548c2a-1484-4966-a9b9-abb8c70ae652","Type":"ContainerStarted","Data":"806b95634ab00bd4bc204529c14100283fe9f81c85249bfb0efb6d1a4c85bacd"}
Nov 03 15:30:16.726579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.726469    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podStartSLOduration=1.726449348 podStartE2EDuration="1.726449348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:16.721260251 +0000 UTC m=+1331.646078166" watchObservedRunningTime="2026-11-03 15:30:16.726449348 +0000 UTC m=+1331.651267267"
Nov 03 15:30:16.749133 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.749103    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:16.822993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:16.821994    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:17.191497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.191463    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"]
Nov 03 15:30:17.195716 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:17.195092    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8fce7d_b2e5_4eba_b73f_a5397b6830b5.slice/crio-602f0eb008831a139670763aa2dcad671d7ba692633848b78001384699b7d968 WatchSource:0}: Error finding container 602f0eb008831a139670763aa2dcad671d7ba692633848b78001384699b7d968: Status 404 returned error can't find the container with id 602f0eb008831a139670763aa2dcad671d7ba692633848b78001384699b7d968
Nov 03 15:30:17.198748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.198521    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:17.198748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.198681    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:17.198920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.198756    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:17.293944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.293918    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"]
Nov 03 15:30:17.297960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.297939    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334461    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334507    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334659    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334691    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334769    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334825    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.334868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334866    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.335115 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334916    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.335115 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.334942    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcf45\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.400912 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.400881    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"]
Nov 03 15:30:17.437026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.436984    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437216 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437048    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437216 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437091    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcf45\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437280    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437308    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437412    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437444    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437525    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.437587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.437569    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.438389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.438364    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.438492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.438414    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.438717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.438696    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.439013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.438955    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.439637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.439504    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.442102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.442048    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.444147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.444125    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.504036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.501817    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcf45\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.504036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.502258    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token\") pod \"e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") " pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.608670 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.608625    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" event={"ID":"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5","Type":"ContainerStarted","Data":"64b4e621db491592b8557ce01d63ca8cd973e0c5a5092e50c7cb2b98a7afb32b"}
Nov 03 15:30:17.609206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.608697    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" event={"ID":"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5","Type":"ContainerStarted","Data":"602f0eb008831a139670763aa2dcad671d7ba692633848b78001384699b7d968"}
Nov 03 15:30:17.613637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.613609    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:17.615645 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.615619    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"]
Nov 03 15:30:17.620485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.620466    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.639548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.636270    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"]
Nov 03 15:30:17.692366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.656250    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2757\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696205    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696350    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696478    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696518    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696550    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696589    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696681    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.696770    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.703376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.665348    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.727924 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.726050    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"]
Nov 03 15:30:17.749828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.749351    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:17.755640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.755563    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"]
Nov 03 15:30:17.759445 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.759383    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podStartSLOduration=1.75936428 podStartE2EDuration="1.75936428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:17.739657775 +0000 UTC m=+1332.664475689" watchObservedRunningTime="2026-11-03 15:30:17.75936428 +0000 UTC m=+1332.684182195"
Nov 03 15:30:17.799841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.799794    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.799885    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.799941    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcjm\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.799991    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800027    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800052    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800085    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800122    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800197    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800235    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800265    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800315    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800343    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800389    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800483    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800540    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2757\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.800837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800628    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.800837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.800673    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.801296 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.801268    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.801933 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.801905    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.803552 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.803528    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.804114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.804088    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.804770 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.804699    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.805423 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.805380    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.807165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.807141    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.932515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.899328    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:17.968759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.938652    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:17.969244 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.965153    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.969460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.969435    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.969628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.969610    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.969730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.969717    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.984157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.984123    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:17.984510 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.984484    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.008882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.984881    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.009260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.009216    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcjm\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.009484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.009460    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.028042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.993356    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.045079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.993875    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.045277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.021378    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.068458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.981528    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.068458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:17.982143    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.103989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.096086    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.109721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.109650    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.114500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.114226    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcjm\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.120245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.115706    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"]
Nov 03 15:30:18.120245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.118751    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.122380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.121310    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2757\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757\") pod \"e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") " pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:18.124180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.124154    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.178459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.178313    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"]
Nov 03 15:30:18.206482 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.206456    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"]
Nov 03 15:30:18.208367 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:18.208340    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76de34ed_49b4_405e_8161_978f3af6c0f2.slice/crio-f55dc3174d57d0e6418ee7be239f6eedaccfc0e40ea7005b0f21e6a67839f926 WatchSource:0}: Error finding container f55dc3174d57d0e6418ee7be239f6eedaccfc0e40ea7005b0f21e6a67839f926: Status 404 returned error can't find the container with id f55dc3174d57d0e6418ee7be239f6eedaccfc0e40ea7005b0f21e6a67839f926
Nov 03 15:30:18.211061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.211030    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.211128 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.211103    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.211164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.211134    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.216770 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.216747    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.216844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.216787    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.216886 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.216874    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.216935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.216924    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.217012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.216944    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hs28\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.217110 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.217089    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.217153 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.217140    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.217305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.217282    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.217371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.217351    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.318626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318594    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.318762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318660    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.318762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318746    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.318869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318793    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.318869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318859    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318907    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.318995    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319021    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319038    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319063    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hs28\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319073    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319462    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319491    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.319845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.319822    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.321329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.321305    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.321633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.321616    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.361050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.354293    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:18.371212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.371182    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:18.392737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.392633    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hs28\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.394278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.394216    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token\") pod \"e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") " pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.450774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.447294    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:18.624684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.624647    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" event={"ID":"76de34ed-49b4-405e-8161-978f3af6c0f2","Type":"ContainerStarted","Data":"5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf"}
Nov 03 15:30:18.625230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.624707    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" event={"ID":"76de34ed-49b4-405e-8161-978f3af6c0f2","Type":"ContainerStarted","Data":"f55dc3174d57d0e6418ee7be239f6eedaccfc0e40ea7005b0f21e6a67839f926"}
Nov 03 15:30:18.650040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.627305    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:18.650040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.627387    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:30:18.752049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.751757    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:18.752049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.751816    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:18.867874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.867817    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.868110 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.867907    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.868110 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.867953    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.868294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.868270    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"]
Nov 03 15:30:18.873753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.873729    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"]
Nov 03 15:30:18.880410 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:18.880358    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620d28d1_d950_453f_9811_0d8f13dd5726.slice/crio-3a9f1866216de1821838847f1a867b5344eb317c0843fac9cfbebd9ee5cbe48f WatchSource:0}: Error finding container 3a9f1866216de1821838847f1a867b5344eb317c0843fac9cfbebd9ee5cbe48f: Status 404 returned error can't find the container with id 3a9f1866216de1821838847f1a867b5344eb317c0843fac9cfbebd9ee5cbe48f
Nov 03 15:30:18.888950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.888897    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.891129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.890588    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.891129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.890657    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:18.970897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.970822    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podStartSLOduration=1.9708003440000001 podStartE2EDuration="1.970800344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:18.947387885 +0000 UTC m=+1333.872205835" watchObservedRunningTime="2026-11-03 15:30:18.970800344 +0000 UTC m=+1333.895618261"
Nov 03 15:30:18.976191 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:18.976151    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"]
Nov 03 15:30:19.104013 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:19.103944    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ce1ad7_d9f4_425d_b94b_fc78ebd7cf55.slice/crio-4f2c457e1ed607ac856dd1e5f91d5caf990cddd9cb469cc13e07442c7ab6ed3c WatchSource:0}: Error finding container 4f2c457e1ed607ac856dd1e5f91d5caf990cddd9cb469cc13e07442c7ab6ed3c: Status 404 returned error can't find the container with id 4f2c457e1ed607ac856dd1e5f91d5caf990cddd9cb469cc13e07442c7ab6ed3c
Nov 03 15:30:19.149065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.120491    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:19.149065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.120595    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:19.149065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.120635    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:19.670082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.638550    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:19.670082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.638606    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:19.711275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.711013    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:19.803996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.753944    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:19.803996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.754026    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:19.820245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.820085    4955 generic.go:356] "Generic (PLEG): container finished" podID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerID="a62aaaf82311f48b3c86b526a65e9801af3454bc8a37507f7d6568d463606d54" exitCode=0
Nov 03 15:30:19.820245 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.820155    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" event={"ID":"36e4d3f5-5f62-4faf-97a5-0bd128a5dded","Type":"ContainerDied","Data":"a62aaaf82311f48b3c86b526a65e9801af3454bc8a37507f7d6568d463606d54"}
Nov 03 15:30:19.822546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.822482    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" event={"ID":"7872f25b-a49f-4d04-8f3f-86de12e1811d","Type":"ContainerStarted","Data":"b940b75de1366a9857faa45de52faab7479c2825903ddc90f50c02606252b8c7"}
Nov 03 15:30:19.827147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.826952    4955 generic.go:356] "Generic (PLEG): container finished" podID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerID="7fa573229d73c8b62df6d0f296a775fae097b8d4e0053ceb56f475b437fd25d7" exitCode=0
Nov 03 15:30:19.827147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.827079    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" event={"ID":"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0","Type":"ContainerDied","Data":"7fa573229d73c8b62df6d0f296a775fae097b8d4e0053ceb56f475b437fd25d7"}
Nov 03 15:30:19.832148 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.832117    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" event={"ID":"620d28d1-d950-453f-9811-0d8f13dd5726","Type":"ContainerStarted","Data":"3a9f1866216de1821838847f1a867b5344eb317c0843fac9cfbebd9ee5cbe48f"}
Nov 03 15:30:19.835189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.835165    4955 generic.go:356] "Generic (PLEG): container finished" podID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerID="d143ac404c40e9b6e3d7f4a96f54df0b083f2e3b7377d7cee637a35dd44b9979" exitCode=0
Nov 03 15:30:19.835431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.835413    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" event={"ID":"d61315aa-8eeb-4073-b7e9-b7759356383c","Type":"ContainerDied","Data":"d143ac404c40e9b6e3d7f4a96f54df0b083f2e3b7377d7cee637a35dd44b9979"}
Nov 03 15:30:19.844926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.844900    4955 generic.go:356] "Generic (PLEG): container finished" podID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerID="642b2deb22f534b49631d4e4fdcc5d855243bba56baa35e28ef220091d29a321" exitCode=0
Nov 03 15:30:19.845539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.845518    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" event={"ID":"478b7585-c756-47b3-bf5f-9aa1790c30a5","Type":"ContainerDied","Data":"642b2deb22f534b49631d4e4fdcc5d855243bba56baa35e28ef220091d29a321"}
Nov 03 15:30:19.878685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:19.850154    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" event={"ID":"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55","Type":"ContainerStarted","Data":"4f2c457e1ed607ac856dd1e5f91d5caf990cddd9cb469cc13e07442c7ab6ed3c"}
Nov 03 15:30:20.291926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.291887    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:30:20.409265 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.362479    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.409613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409580    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.409741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409618    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.409741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409649    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.409853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409768    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.409853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409803    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.409853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409838    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.410019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.409900    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.410112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.410095    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs\") pod \"d61315aa-8eeb-4073-b7e9-b7759356383c\" (UID: \"d61315aa-8eeb-4073-b7e9-b7759356383c\") "
Nov 03 15:30:20.411684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.411319    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:30:20.413096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.413056    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.413096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.413068    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.413267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.413122    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.413373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.413325    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.413558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.413537    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:20.422603 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.422560    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "kube-api-access-txzdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:20.422844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.422802    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:30:20.424004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.423939    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy" pod "d61315aa-8eeb-4073-b7e9-b7759356383c" (UID: "d61315aa-8eeb-4073-b7e9-b7759356383c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531192    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531224    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531238    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531250    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531267    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzdc\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-kube-api-access-txzdc\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531281    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61315aa-8eeb-4073-b7e9-b7759356383c-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531294    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531308    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61315aa-8eeb-4073-b7e9-b7759356383c-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.579005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.531321    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61315aa-8eeb-4073-b7e9-b7759356383c-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:20.715013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.679359    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:20.715013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.679420    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:20.829176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.828689    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:30:20.848543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.848448    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:30:20.898004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.894306    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:20.898004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.894370    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:20.909623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.908398    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc" event={"ID":"d61315aa-8eeb-4073-b7e9-b7759356383c","Type":"ContainerDied","Data":"375dc1793c837335c12a310b4b322f82b1473f2847d09fbe0dec3f238fab9050"}
Nov 03 15:30:20.909623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.908474    4955 scope.go:122] "RemoveContainer" containerID="d143ac404c40e9b6e3d7f4a96f54df0b083f2e3b7377d7cee637a35dd44b9979"
Nov 03 15:30:20.909623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.908638    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914577    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914663    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914742    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914780    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914838    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914863    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914941    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.914995    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915031    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915071    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915127    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915273    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915300    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo\") pod \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\" (UID: \"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915334    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915380    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915476    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915518    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.915584    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs\") pod \"478b7585-c756-47b3-bf5f-9aa1790c30a5\" (UID: \"478b7585-c756-47b3-bf5f-9aa1790c30a5\") "
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.916804    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.917063    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.917265    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.917582    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.918433    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.919362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.918737    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:30:20.921236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.921048    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:30:20.921542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.921380    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.921712 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.921659    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.924638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.923744    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.924638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.924232    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt" event={"ID":"478b7585-c756-47b3-bf5f-9aa1790c30a5","Type":"ContainerDied","Data":"30cd64ece9c6a40ff072d9821f8379a81cd6c5766dfb9f0ba4899225d745a033"}
Nov 03 15:30:20.924638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.924356    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"
Nov 03 15:30:20.925205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.924880    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:20.926370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.926344    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:20.927035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.926930    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:30:21.008086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.927549    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" event={"ID":"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55","Type":"ContainerStarted","Data":"cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949"}
Nov 03 15:30:21.008086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:20.962844    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" event={"ID":"7872f25b-a49f-4d04-8f3f-86de12e1811d","Type":"ContainerStarted","Data":"9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c"}
Nov 03 15:30:21.008086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.007315    4955 scope.go:122] "RemoveContainer" containerID="642b2deb22f534b49631d4e4fdcc5d855243bba56baa35e28ef220091d29a321"
Nov 03 15:30:21.010140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.010101    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz" event={"ID":"47a8f15d-2ed8-47b0-9919-9506b5a1f0a0","Type":"ContainerDied","Data":"7668b94140bcef11073fdde12ec60e0d353c8477fd0efc749c46841d5016e70b"}
Nov 03 15:30:21.010450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.010431    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"
Nov 03 15:30:21.014187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.012889    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "kube-api-access-bg6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:21.014187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.012950    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy" pod "478b7585-c756-47b3-bf5f-9aa1790c30a5" (UID: "478b7585-c756-47b3-bf5f-9aa1790c30a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:21.014187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.013440    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:30:21.014187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.013852    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "kube-api-access-nwmh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:21.014187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.014032    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token" pod "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" (UID: "47a8f15d-2ed8-47b0-9919-9506b5a1f0a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:21.019400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.018390    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" event={"ID":"620d28d1-d950-453f-9811-0d8f13dd5726","Type":"ContainerStarted","Data":"64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074"}
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020262    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478b7585-c756-47b3-bf5f-9aa1790c30a5-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020288    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020302    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020316    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020330    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020345    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020358    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020372    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020386    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020399    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020415    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmh9\" (UniqueName: \"kubernetes.io/projected/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-kube-api-access-nwmh9\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020429    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020444    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020458    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478b7585-c756-47b3-bf5f-9aa1790c30a5-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020471    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020484    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020498    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6kd\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-kube-api-access-bg6kd\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.022535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.020511    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478b7585-c756-47b3-bf5f-9aa1790c30a5-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.116877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.105820    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:30:21.116877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.113440    4955 scope.go:122] "RemoveContainer" containerID="7fa573229d73c8b62df6d0f296a775fae097b8d4e0053ceb56f475b437fd25d7"
Nov 03 15:30:21.226255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226213    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226285    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226320    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226381    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226456    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226493    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226598    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226681    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:21.226721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226698    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226855 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226835    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert\") pod \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\" (UID: \"36e4d3f5-5f62-4faf-97a5-0bd128a5dded\") "
Nov 03 15:30:21.226913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.226869    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:21.227099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.227048    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:21.227928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.227682    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:21.228066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.227945    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.228066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.227963    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.228066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.228033    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.228066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.228045    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.228490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.228462    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:30:21.238026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.237988    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:30:21.238747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.238720    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "kube-api-access-6bql7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:21.246678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.246623    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:21.259577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.259519    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy" pod "36e4d3f5-5f62-4faf-97a5-0bd128a5dded" (UID: "36e4d3f5-5f62-4faf-97a5-0bd128a5dded"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:30:21.281377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.281308    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"]
Nov 03 15:30:21.304416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.304364    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-s7qpc"]
Nov 03 15:30:21.329687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.329569    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bql7\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-kube-api-access-6bql7\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.329687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.329616    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.329687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.329633    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.329687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.329646    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.329687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.329660    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e4d3f5-5f62-4faf-97a5-0bd128a5dded-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:21.375425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.374904    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:21.375425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.375059    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:21.375425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.375103    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:21.376926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.376907    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:21.377212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.377147    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:21.377330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.377311    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:21.512086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.485575    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:21.647397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.644285    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:21.647397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.644349    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:21.647397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.645771    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:21.647397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.645815    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:21.745000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.742886    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podStartSLOduration=4.742863962 podStartE2EDuration="4.742863962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:21.732731309 +0000 UTC m=+1336.657549222" watchObservedRunningTime="2026-11-03 15:30:21.742863962 +0000 UTC m=+1336.667681875"
Nov 03 15:30:21.754451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.754411    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:21.754700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:21.754672    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:22.081050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.076454    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podStartSLOduration=5.076431317 podStartE2EDuration="5.076431317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:21.923335016 +0000 UTC m=+1336.848152930" watchObservedRunningTime="2026-11-03 15:30:22.076431317 +0000 UTC m=+1337.001249231"
Nov 03 15:30:22.354208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.354079    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podStartSLOduration=5.35405597 podStartE2EDuration="5.35405597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:22.230364383 +0000 UTC m=+1337.155182308" watchObservedRunningTime="2026-11-03 15:30:22.35405597 +0000 UTC m=+1337.278873883"
Nov 03 15:30:22.377012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.373670    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:22.377012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.373743    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:22.389904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.389726    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:22.389904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.389784    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:22.511141 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.511088    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:22.511864 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.511833    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": context deadline exceeded"
Nov 03 15:30:22.589995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.542927    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:22.589995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.543514    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:22.703065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.635152    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:22.703065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.641179    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.013058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.928888    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.013058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:22.928951    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:23.087173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.087127    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.087580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.087550    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.114008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.106774    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.121014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.112659    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4" event={"ID":"36e4d3f5-5f62-4faf-97a5-0bd128a5dded","Type":"ContainerDied","Data":"468c8d0feb6d694b027aece67c9a3dc6fb5c839e8572b905b2c6154930726378"}
Nov 03 15:30:23.121014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.114472    4955 scope.go:122] "RemoveContainer" containerID="a62aaaf82311f48b3c86b526a65e9801af3454bc8a37507f7d6568d463606d54"
Nov 03 15:30:23.121014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.112819    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"
Nov 03 15:30:23.260082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.221837    4955 scope.go:122] "RemoveContainer" containerID="72249dc53c7404c96c89253ed8bda8b24548992d082f90c5c946dae310cef6ba"
Nov 03 15:30:23.369028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.346822    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.480995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.462279    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:23.522045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.487210    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:23.558210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.546065    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.579996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.576743    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:23.703390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.703336    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:30:23.715542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.710458    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.715542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.710527    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:23.738058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.723087    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.738058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.723168    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:23.738058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.725417    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.738058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.725464    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:23.738058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.726274    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.738058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.726311    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:23.900003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.899487    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.900003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.899561    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:23.900273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.899931    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:23.900273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.900062    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:23.965046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.947591    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" path="/var/lib/kubelet/pods/d61315aa-8eeb-4073-b7e9-b7759356383c/volumes"
Nov 03 15:30:23.998832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:23.998792    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:30:24.064021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.045193    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"]
Nov 03 15:30:24.064021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.045278    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:30:24.085002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.081904    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-nhqvz"]
Nov 03 15:30:24.348026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.345268    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.389224    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.389661    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"]
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.389747    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.389790    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.390617    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.391763    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.407296    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.407353    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.433592    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:24.435548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.433649    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:24.443210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.442563    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-njfkt"]
Nov 03 15:30:24.451178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.450617    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:24.451178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.450672    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:24.496717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.494317    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.496717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.494697    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:30:24.496717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.495566    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.496717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.495892    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:30:24.587284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.586684    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.614992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.609493    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.614992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.609740    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:30:24.634037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.624329    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.643054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.639653    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:24.650005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.645045    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:24.719017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.710583    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.773729 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.760072    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:24.785443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.784165    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:24.785443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.784322    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:25.004998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:24.988235    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.313519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.313483    4955 scope.go:122] "RemoveContainer" containerID="abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f"
Nov 03 15:30:25.396088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.396044    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:25.397867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.397834    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.399168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.399135    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.426569 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.426531    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:25.426833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.426802    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:25.467307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.464683    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:25.467307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.464738    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:25.467307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.465017    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:25.467307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.465052    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:25.536996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.522311    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:25.536996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.521409    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.589083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.588257    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.627995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.620229    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.627995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.626137    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.712193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.712085    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:25.754998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.723062    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:25.754998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.723120    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:25.806996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.773737    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:25.806996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.773825    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:25.968089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.964284    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" path="/var/lib/kubelet/pods/478b7585-c756-47b3-bf5f-9aa1790c30a5/volumes"
Nov 03 15:30:25.969083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:25.968735    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" path="/var/lib/kubelet/pods/47a8f15d-2ed8-47b0-9919-9506b5a1f0a0/volumes"
Nov 03 15:30:26.500227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.498516    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:26.542704 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:26.542590    4955 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.249s"
Nov 03 15:30:26.546870 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.546842    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:30:26.547352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.547331    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"]
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.666066    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.666145    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.687988    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.688052    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.714692    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.718514    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.718790    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.719148    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:26.726546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.719183    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:26.742198 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.742151    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:26.835032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.834987    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:26.836807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.836775    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:26.839280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.839254    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:26.916640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:26.916599    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-bmbr4"]
Nov 03 15:30:27.225999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.203525    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.225999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.203595    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:27.378998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.371357    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:27.491056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.419808    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.491056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.419881    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:27.493504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.493467    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.496995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.493750    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:27.616692 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.501201    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:27.777065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.522052    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.777065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.776771    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:27.777065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.524144    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.777065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.776825    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:27.777065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.636040    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.777065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.776861    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:27.834050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.827709    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:27.834050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.833148    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:30:27.861998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.837091    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg" podUID="45c5a8e6-be22-4a5f-862e-c7d47d36f39a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:27.861998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.846485    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:27.861998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.855385    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 15:30:27.861998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.855432    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 15:30:27.878254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.878196    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:30:27.962195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.929307    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:27.962195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.930421    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:27.962195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:27.950030    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.321336    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" path="/var/lib/kubelet/pods/36e4d3f5-5f62-4faf-97a5-0bd128a5dded/volumes"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.321848    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.321926    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"]
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322821    4955 cpu_manager.go:414] "containerMap: removing container" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322841    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322868    4955 cpu_manager.go:414] "containerMap: removing container" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322876    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322918    4955 cpu_manager.go:414] "containerMap: removing container" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy"
Nov 03 15:30:28.354025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322926    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy"
Nov 03 15:30:28.408037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.322960    4955 cpu_manager.go:414] "containerMap: removing container" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy"
Nov 03 15:30:28.408300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.408278    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy"
Nov 03 15:30:28.408671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.408648    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a8f15d-2ed8-47b0-9919-9506b5a1f0a0" containerName="istio-proxy"
Nov 03 15:30:28.408834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.408821    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e4d3f5-5f62-4faf-97a5-0bd128a5dded" containerName="istio-proxy"
Nov 03 15:30:28.408945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.408934    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61315aa-8eeb-4073-b7e9-b7759356383c" containerName="istio-proxy"
Nov 03 15:30:28.409083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.409071    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="478b7585-c756-47b3-bf5f-9aa1790c30a5" containerName="istio-proxy"
Nov 03 15:30:28.420046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.383243    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:28.425628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.382484    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:28.425881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.425849    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:28.453255 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:28.453196    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.455277    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.489120    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.462673    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.489206    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.514305    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"]
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.514469    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.516951    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517018    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517171    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517205    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517262    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517302    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517466    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6xh\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517535    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.522996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.517563    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.619065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.577690    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cceaf27f-openshift-default-dockercfg-b8jlf\""
Nov 03 15:30:28.620270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.620226    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 15:30:28.620479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.620444    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 15:30:28.648728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.648690    4955 generic.go:356] "Generic (PLEG): container finished" podID="4f91448a-36f7-497d-b013-0efa7997b82d" containerID="efffa6ed60e8f3b2f196bfd9b3de420cccd8b524e9dbdee8f1a25c2fcee73178" exitCode=60
Nov 03 15:30:28.667878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.667065    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" event={"ID":"4f91448a-36f7-497d-b013-0efa7997b82d","Type":"ContainerDied","Data":"efffa6ed60e8f3b2f196bfd9b3de420cccd8b524e9dbdee8f1a25c2fcee73178"}
Nov 03 15:30:28.668235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.668210    4955 scope.go:122] "RemoveContainer" containerID="72249dc53c7404c96c89253ed8bda8b24548992d082f90c5c946dae310cef6ba"
Nov 03 15:30:28.694769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.694745    4955 scope.go:122] "RemoveContainer" containerID="efffa6ed60e8f3b2f196bfd9b3de420cccd8b524e9dbdee8f1a25c2fcee73178"
Nov 03 15:30:28.695525 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:28.695499    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56_odh-ai-gateway-infra(4f91448a-36f7-497d-b013-0efa7997b82d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" podUID="4f91448a-36f7-497d-b013-0efa7997b82d"
Nov 03 15:30:28.772359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.772305    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.773113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.773076    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.773543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.773523    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6xh\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.773760 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.773743    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.773912 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.773896    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.774100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.774071    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.774288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.774256    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.775295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.775267    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.816804 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.816773    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.902745 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.815214    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.949577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.816699    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.949809 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.817835    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.949990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.843151    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.950211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.844505    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:28.951370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.844591    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:28.961844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.797126    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.962084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.890872    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.968279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.891508    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:30:28.976371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.891542    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:28.978472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.891572    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:30:28.979148 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.891614    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:28.979535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.901492    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.998315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.915478    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6xh\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:28.998553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.942826    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:28.999015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:28.958652    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:29.589105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.581757    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:29.617007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.591770    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token\") pod \"e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") " pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:29.726084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.669136    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:29.726676 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.726641    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:29.776366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.743315    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:29.776366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.743382    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:29.915114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.902220    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:29.965053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:29.964089    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:30.009178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.006804    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:30.009178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.008038    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:30.112012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.018120    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:30.112012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.018260    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:30:30.112012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.032107    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:30.112012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.054125    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:30:30.112012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.061750    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:30:30.175046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.149209    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:30.596946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.596902    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:30.597444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.235706    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:30:30.597662 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.597649    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:30.598916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.588024    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:30.600783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.599870    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:30.617013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.613677    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"]
Nov 03 15:30:30.617013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.614360    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:30.617013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.614420    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:30.621362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.621333    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:30.654049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.623183    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:30.666847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.666807    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:30.702619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.686481    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:30.702619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.686562    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:30.702619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.687096    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:30.702619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.687136    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:30.719322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.719265    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:30:30.807343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.807313    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:30:30.814989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.810021    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:30:30.825462 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:30.823221    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerStarted","Data":"7ce6d602b71a7980501039271be55804aea5c05176b72ba2ea022a576c96f19a"}
Nov 03 15:30:31.007752 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.007719    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:30:31.008030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.008013    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:30:31.022112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.015828    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:30:31.022112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.017027    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:30:31.022112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.019634    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:31.022112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.021929    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:31.056993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.056066    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:31.079129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.068186    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:31.082988 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.082071    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:30:31.096550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.096517    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:30:31.101314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.101250    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:30:31.177055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.158291    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:30:31.180623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.180589    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:30:31.380175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.380134    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:31.380522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.380489    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:31.381002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.380956    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 15:30:31.381155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.381135    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 15:30:31.451039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.445407    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:31.451039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.447753    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:31.451039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.447800    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:31.492564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.492505    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:31.493273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.492852    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:31.493273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.492964    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:31.495177 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.494746    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"]
Nov 03 15:30:31.634395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.632031    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln\" (UniqueName: \"kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln\") pod \"4f91448a-36f7-497d-b013-0efa7997b82d\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") "
Nov 03 15:30:31.634395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.632141    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca\") pod \"4f91448a-36f7-497d-b013-0efa7997b82d\" (UID: \"4f91448a-36f7-497d-b013-0efa7997b82d\") "
Nov 03 15:30:31.634395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.633646    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca" pod "4f91448a-36f7-497d-b013-0efa7997b82d" (UID: "4f91448a-36f7-497d-b013-0efa7997b82d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:30:31.638227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.638144    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln" pod "4f91448a-36f7-497d-b013-0efa7997b82d" (UID: "4f91448a-36f7-497d-b013-0efa7997b82d"). InnerVolumeSpecName "kube-api-access-tvfln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:31.750528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.744123    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvfln\" (UniqueName: \"kubernetes.io/projected/4f91448a-36f7-497d-b013-0efa7997b82d-kube-api-access-tvfln\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:31.750528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.744158    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91448a-36f7-497d-b013-0efa7997b82d-maas-api-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:31.983197 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.983075    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56" event={"ID":"4f91448a-36f7-497d-b013-0efa7997b82d","Type":"ContainerDied","Data":"a94b759df2e231ae1d6fb5b29dfcdbb9f64d86cf9423d21848efcf4fa3d3d3ed"}
Nov 03 15:30:31.983846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.983822    4955 scope.go:122] "RemoveContainer" containerID="efffa6ed60e8f3b2f196bfd9b3de420cccd8b524e9dbdee8f1a25c2fcee73178"
Nov 03 15:30:31.984208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:31.984189    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"
Nov 03 15:30:32.131533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.131462    4955 generic.go:356] "Generic (PLEG): container finished" podID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerID="7ce6d602b71a7980501039271be55804aea5c05176b72ba2ea022a576c96f19a" exitCode=60
Nov 03 15:30:32.134085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.134055    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerDied","Data":"7ce6d602b71a7980501039271be55804aea5c05176b72ba2ea022a576c96f19a"}
Nov 03 15:30:32.134757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.134737    4955 scope.go:122] "RemoveContainer" containerID="7ce6d602b71a7980501039271be55804aea5c05176b72ba2ea022a576c96f19a"
Nov 03 15:30:32.135459 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:32.135432    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895330-hxgvx_odh-ai-gateway-infra(44880eaf-aec3-45b9-9b28-3930d6a4bad4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4"
Nov 03 15:30:32.156315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.150986    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" event={"ID":"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde","Type":"ContainerStarted","Data":"00666d865c28747ba92bf47e979ff60178ec570f55706c8a95e1bf3d40cf40f9"}
Nov 03 15:30:32.503999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.495034    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 15:30:32.503999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.495101    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 15:30:32.523439 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.523404    4955 scope.go:122] "RemoveContainer" containerID="abe36470f514ff0a359ac32f48c6151f49e9a4a235ed2c7b868cc3d67d61da0f"
Nov 03 15:30:32.594446 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.566263    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:32.594446 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:32.566329    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:33.631999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.597306    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lhbx9" podUID="44c5a3f3-e5c3-4f67-99de-5eb55e05bd06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:33.683424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.674525    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 15:30:33.683424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.674589    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 15:30:33.825019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.821884    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:33.825019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.821942    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:33.854032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.850368    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:33.868149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.860063    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:33.954410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.860108    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:30:33.981107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.871494    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:33.981550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.981531    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:34.002057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:33.960658    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:30:34.002353 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.002310    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:30:34.011102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.011058    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:34.027152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.022082    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:34.049181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.022690    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:30:34.049717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.049664    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" event={"ID":"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde","Type":"ContainerStarted","Data":"78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"}
Nov 03 15:30:34.050060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.050045    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:34.076262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.022718    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:30:34.193560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.193527    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"]
Nov 03 15:30:34.404996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.401391    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:34.429181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.429139    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:34.429457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.429427    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:34.575064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.566111    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:34.876046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.775678    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:34.876046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.777023    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:34.876571 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.876161    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:34.920999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.876784    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:30:34.920999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.917057    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Nov 03 15:30:34.943995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.933086    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:34.943995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.933157    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.014007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.944440    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:30:35.014007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.944501    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:30:35.014007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.968566    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.014007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.981127    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:30:35.014007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.981229    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.014007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:34.982631    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:30:35.044013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.042034    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.044013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.042576    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.044013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.042916    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:30:35.044013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.043000    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:30:35.181997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.156706    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.182698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.182667    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:30:35.406048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.406007    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.515055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.507233    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:35.878999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.594065    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.881126 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.881072    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:35.891176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.554941    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:35.932122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.928578    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:30:35.978426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:35.940889    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:36.118070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.001481    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:36.142996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.024223    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:36.142996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.040826    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:36.377339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.305500    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:30:36.377339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.331107    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:36.377339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.362615    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:36.417085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.414705    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:36.607997 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:36.564046    4955 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.337s"
Nov 03 15:30:36.607997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.565638    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:30:36.607997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.565680    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"]
Nov 03 15:30:36.607997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.565713    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-34e8c5-29895330-q4n56"]
Nov 03 15:30:36.607997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.565849    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:36.953058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.565958    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:30:36.953058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.940216    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:36.953058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.940323    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:30:36.953058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.940636    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:36.954157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.954114    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:37.050153 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.954326    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:37.094332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.657282    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" podUID="a4848f69-a0a9-4007-9203-7c656b86ef62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:37.121171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.121094    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:30:37.384780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.923545    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:37.409899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:36.923495    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podStartSLOduration=11.923471122 podStartE2EDuration="11.923471122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:34.596496593 +0000 UTC m=+1349.521314509" watchObservedRunningTime="2026-11-03 15:30:36.923471122 +0000 UTC m=+1351.848289035"
Nov 03 15:30:37.431335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.151456    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:37.459007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.458940    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:37.933251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.151544    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:37.934589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.314075    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:30:37.935335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.314443    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:37.935491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.361367    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:37.935637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.935610    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:37.935752 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.602104    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:37.935850 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.151521    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Nov 03 15:30:37.936001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:37.441164    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:38.128613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.128564    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:30:38.174994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.150132    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:38.464335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.464294    4955 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-x66cq container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:38.464611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.464579    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-x66cq" podUID="a4e32724-e514-41cc-b175-59da7bfd7b8b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:38.541996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.516498    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:30:38.701997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.524110    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" podUID="a4848f69-a0a9-4007-9203-7c656b86ef62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 03 15:30:38.701997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.576037    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:38.701997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:38.645431    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:39.006226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.006183    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.006618 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.006591    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.006888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.006866    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.529996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.527581    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:39.675221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.667730    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.936787    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.945300    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg" podUID="45c5a8e6-be22-4a5f-862e-c7d47d36f39a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.947164    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.962351    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.969541    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.969809    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.970011    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:39.970649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:39.970041    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:40.133138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.133102    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:40.139056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.135064    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:40.220108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.220060    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:40.220708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.220636    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:40.221103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.221081    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:30:40.513055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.451514    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" path="/var/lib/kubelet/pods/4f91448a-36f7-497d-b013-0efa7997b82d/volumes"
Nov 03 15:30:40.855046 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:40.852362    4955 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.25s"
Nov 03 15:30:40.871095 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.627153    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Nov 03 15:30:40.871095 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.756355    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.064991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.036778    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:30:41.080254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.080209    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.080822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.080797    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:30:41.081210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.081188    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:30:41.193030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.627204    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:41.206002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.193297    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.226994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.193463    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.255012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.246928    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:30:41.259643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:40.866268    4955 trace.go:236] Trace[1903022811]: "Calculate volume metrics of istio-csr-dns-cert for pod openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmjhb" (03-Nov-2026 15:30:39.118) (total time: 1748ms):
Nov 03 15:30:41.259643 ip-10-0-134-23 kubenswrapper[4955]: Trace[1903022811]: [1.748050207s] [1.748050207s] END
Nov 03 15:30:41.262019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.081548    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:41.262284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.262250    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.363720 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.320916    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m" pod "44880eaf-aec3-45b9-9b28-3930d6a4bad4" (UID: "44880eaf-aec3-45b9-9b28-3930d6a4bad4"). InnerVolumeSpecName "kube-api-access-rhn8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:41.364019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.253274    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m\" (UniqueName: \"kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m\") pod \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") "
Nov 03 15:30:41.364383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.364355    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca\") pod \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\" (UID: \"44880eaf-aec3-45b9-9b28-3930d6a4bad4\") "
Nov 03 15:30:41.365451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.365427    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhn8m\" (UniqueName: \"kubernetes.io/projected/44880eaf-aec3-45b9-9b28-3930d6a4bad4-kube-api-access-rhn8m\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:41.365572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.252357    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:30:41.379960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.252294    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:30:41.383636 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.383601    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca" pod "44880eaf-aec3-45b9-9b28-3930d6a4bad4" (UID: "44880eaf-aec3-45b9-9b28-3930d6a4bad4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:30:41.383837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.252337    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.393959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.393925    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:41.394141 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.394005    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.503054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.501803    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44880eaf-aec3-45b9-9b28-3930d6a4bad4-maas-api-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:41.775074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.736233    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:41.868960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:41.868906    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:42.081147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.081031    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:42.081887 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.081383    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:30:42.082263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.081450    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:42.082554 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.081473    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:30:42.090310 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.090264    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx" event={"ID":"44880eaf-aec3-45b9-9b28-3930d6a4bad4","Type":"ContainerDied","Data":"04d23112ec5ecc97ef445b40b6b3d27acfde1fc0cb84b890c823acdd52770c37"}
Nov 03 15:30:42.090522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.090505    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:30:42.090672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.090645    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:42.090841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.090827    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:30:42.090986 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.090957    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:30:42.091127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.091116    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:30:42.102677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.091922    4955 scope.go:122] "RemoveContainer" containerID="7ce6d602b71a7980501039271be55804aea5c05176b72ba2ea022a576c96f19a"
Nov 03 15:30:42.170156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.169571    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:30:42.246167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.246076    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 03 15:30:42.257619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.256309    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:42.322700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.310099    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:30:42.322700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.310164    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:42.371429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.371332    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:30:42.390447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.389638    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:30:42.505049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.505015    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7307-449bd7"]
Nov 03 15:30:42.517907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.506129    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:42.527008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.518137    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:42.527268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527243    4955 cpu_manager.go:414] "containerMap: removing container" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.527364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527351    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.527487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527476    4955 cpu_manager.go:414] "containerMap: removing container" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.527558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527549    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.527647 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527636    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:42.527723 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527712    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:42.530000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.527807    4955 cpu_manager.go:414] "containerMap: removing container" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.530170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.530150    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.530498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.530482    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.580096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.580066    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:42.584063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.584028    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:42.584252 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.584241    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679215    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679266    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679285    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7307-449bd7"]
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679324    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679360    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679393    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.679427    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.681824    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:42.689098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.686702    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:42.690520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.690075    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:30:42.733551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.733513    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"]
Nov 03 15:30:42.739176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.739126    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895330-hxgvx"]
Nov 03 15:30:42.775042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.774964    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvkj\" (UniqueName: \"kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj\") pod \"test-curl-7307-449bd7\" (UID: \"16148679-89ee-4ff9-8a42-6df331ae36ae\") " pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:42.808455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.808421    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hb74t\""
Nov 03 15:30:42.877800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.877768    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvkj\" (UniqueName: \"kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj\") pod \"test-curl-7307-449bd7\" (UID: \"16148679-89ee-4ff9-8a42-6df331ae36ae\") " pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:42.944711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:42.944637    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvkj\" (UniqueName: \"kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj\") pod \"test-curl-7307-449bd7\" (UID: \"16148679-89ee-4ff9-8a42-6df331ae36ae\") " pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:43.026029 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:43.025959    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:43.218805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:43.218721    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" path="/var/lib/kubelet/pods/44880eaf-aec3-45b9-9b28-3930d6a4bad4/volumes"
Nov 03 15:30:43.454375 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:43.454339    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16148679_89ee_4ff9_8a42_6df331ae36ae.slice/crio-1c8675d32d58b80c9d19f80e4c0825328ee8b65a3a9300f3f60d6c823dc10471 WatchSource:0}: Error finding container 1c8675d32d58b80c9d19f80e4c0825328ee8b65a3a9300f3f60d6c823dc10471: Status 404 returned error can't find the container with id 1c8675d32d58b80c9d19f80e4c0825328ee8b65a3a9300f3f60d6c823dc10471
Nov 03 15:30:43.459902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:43.459860    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7307-449bd7"]
Nov 03 15:30:44.236032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:44.235998    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7307-449bd7" event={"ID":"16148679-89ee-4ff9-8a42-6df331ae36ae","Type":"ContainerStarted","Data":"1c8675d32d58b80c9d19f80e4c0825328ee8b65a3a9300f3f60d6c823dc10471"}
Nov 03 15:30:46.895352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.895322    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"]
Nov 03 15:30:46.896722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.896695    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:46.896722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.896719    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:46.896919 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.896888    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f91448a-36f7-497d-b013-0efa7997b82d" containerName="cleanup"
Nov 03 15:30:46.896919 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.896909    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="44880eaf-aec3-45b9-9b28-3930d6a4bad4" containerName="cleanup"
Nov 03 15:30:46.908861 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.908839    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:46.961727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:46.961687    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"]
Nov 03 15:30:47.002951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.002914    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.002951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.002956    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003164    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003301 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003272    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4m2\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003475 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003456    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003541 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003502    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003651 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003618    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003702    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.003767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.003737    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105571 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105534    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105586    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105630    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105664    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105736    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105795    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4m2\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.105987 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105933    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.105995    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.106002    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.106038    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.106099    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.106168    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106482 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.106457    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.106602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.106513    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.108399 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.108378    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.108727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.108704    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.169440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.169365    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.169573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.169480    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4m2\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.227626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.227597    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:47.371196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.371155    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7307-449bd7" event={"ID":"16148679-89ee-4ff9-8a42-6df331ae36ae","Type":"ContainerStarted","Data":"bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35"}
Nov 03 15:30:47.487358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.487238    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7307-449bd7" podStartSLOduration=2.9179370799999997 podStartE2EDuration="6.487207445s" totalImagesPullingTime="3.569270365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:47.486945969 +0000 UTC m=+1362.411763883" watchObservedRunningTime="2026-11-03 15:30:47.487207445 +0000 UTC m=+1362.412025364"
Nov 03 15:30:47.663578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.663549    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"]
Nov 03 15:30:47.664118 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:30:47.664090    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dce39a_4460_4fbb_a3c3_d640b4718a0f.slice/crio-f4526678e2323261088d4aac6ecced05544eef30533c64daa1d7161ed714ecb7 WatchSource:0}: Error finding container f4526678e2323261088d4aac6ecced05544eef30533c64daa1d7161ed714ecb7: Status 404 returned error can't find the container with id f4526678e2323261088d4aac6ecced05544eef30533c64daa1d7161ed714ecb7
Nov 03 15:30:47.666791 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.666760    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:47.666862 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.666828    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:47.666862 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:47.666857    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:30:48.380261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:48.380223    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" event={"ID":"04dce39a-4460-4fbb-a3c3-d640b4718a0f","Type":"ContainerStarted","Data":"31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26"}
Nov 03 15:30:48.380751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:48.380283    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" event={"ID":"04dce39a-4460-4fbb-a3c3-d640b4718a0f","Type":"ContainerStarted","Data":"f4526678e2323261088d4aac6ecced05544eef30533c64daa1d7161ed714ecb7"}
Nov 03 15:30:48.528088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:48.528035    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podStartSLOduration=2.528022069 podStartE2EDuration="2.528022069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:30:48.526478475 +0000 UTC m=+1363.451296411" watchObservedRunningTime="2026-11-03 15:30:48.528022069 +0000 UTC m=+1363.452839981"
Nov 03 15:30:49.228188 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:49.228140    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:49.253999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:49.251576    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 15:30:49.253999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:49.251638    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 15:30:49.270298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:49.270264    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7307-449bd7"]
Nov 03 15:30:49.273762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:49.273734    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7307-449bd7"]
Nov 03 15:30:49.386425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:49.386387    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7307-449bd7" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" containerName="test-curl-7307-449bd7" containerID="cri-o://bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35" gracePeriod=2
Nov 03 15:30:49.450996 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:49.443223    4955 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7307-449bd7\" is forbidden: User \"system:node:ip-10-0-134-23.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-23.us-west-1.compute.internal' and this object" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:50.228820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:50.228786    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 15:30:50.229066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:50.228837    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 15:30:51.228780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:51.228743    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 15:30:51.229270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:51.228810    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 15:30:51.719025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:51.719000    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:51.804980 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:51.804932    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj\" (UniqueName: \"kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj\") pod \"16148679-89ee-4ff9-8a42-6df331ae36ae\" (UID: \"16148679-89ee-4ff9-8a42-6df331ae36ae\") "
Nov 03 15:30:51.807306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:51.807283    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj" pod "16148679-89ee-4ff9-8a42-6df331ae36ae" (UID: "16148679-89ee-4ff9-8a42-6df331ae36ae"). InnerVolumeSpecName "kube-api-access-tvvkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:30:51.907371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:51.907330    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvvkj\" (UniqueName: \"kubernetes.io/projected/16148679-89ee-4ff9-8a42-6df331ae36ae-kube-api-access-tvvkj\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:30:52.228732 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.228639    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 15:30:52.228732 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.228714    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 15:30:52.408377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.408333    4955 generic.go:356] "Generic (PLEG): container finished" podID="16148679-89ee-4ff9-8a42-6df331ae36ae" containerID="bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35" exitCode=137
Nov 03 15:30:52.408377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.408377    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:52.408615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.408431    4955 scope.go:122] "RemoveContainer" containerID="bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35"
Nov 03 15:30:52.418570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.418541    4955 scope.go:122] "RemoveContainer" containerID="bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35"
Nov 03 15:30:52.418911 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:52.418891    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35\": container with ID starting with bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35 not found: ID does not exist" containerID="bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35"
Nov 03 15:30:52.419028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:52.418921    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35"} err="failed to get container status \"bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35\": rpc error: code = NotFound desc = could not find container \"bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35\": container with ID starting with bf40152ad991f7100bb9ac42fb437ccd9c4b1148f38c0eb0f6a5816844ac8f35 not found: ID does not exist"
Nov 03 15:30:52.468730 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:52.468687    4955 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7307-449bd7\" is forbidden: User \"system:node:ip-10-0-134-23.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-23.us-west-1.compute.internal' and this object" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" pod="opendatahub/test-curl-7307-449bd7"
Nov 03 15:30:53.207065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:53.207026    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" path="/var/lib/kubelet/pods/16148679-89ee-4ff9-8a42-6df331ae36ae/volumes"
Nov 03 15:30:53.228412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:53.228385    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 15:30:53.228590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:53.228434    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 15:30:54.228549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:54.228509    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 15:30:54.229088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:54.228578    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 15:30:55.282330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:55.280176    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:55.282330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:55.281154    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:55.282863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:30:55.282473    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:30:57.939790 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:30:57.939748    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:31:01.495007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.492466    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"]
Nov 03 15:31:01.495007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.493916    4955 cpu_manager.go:414] "containerMap: removing container" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" containerName="test-curl-7307-449bd7"
Nov 03 15:31:01.495007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.493938    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" containerName="test-curl-7307-449bd7"
Nov 03 15:31:01.495007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.494240    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="16148679-89ee-4ff9-8a42-6df331ae36ae" containerName="test-curl-7307-449bd7"
Nov 03 15:31:01.502083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.502057    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.611330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.611293    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"]
Nov 03 15:31:01.677559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677519    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.677768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677607    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.677768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677668    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.677768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677694    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.677877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677788    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.678008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677941    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.678008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.677962    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.678101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.678081    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsd28\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.678184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.678169    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.779700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.779610    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.779700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.779660    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.779700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.779689    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.779756    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.779790    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.779857    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780028    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780063    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780111    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsd28\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780150    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780278    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780308    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780514    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.780592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.780574    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.782316 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.782286    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.782533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.782516    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.842748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.842710    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:01.842901 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:01.842761    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsd28\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28\") pod \"e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") " pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:02.115841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:02.115741    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:02.556961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:02.553266    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"]
Nov 03 15:31:02.556961 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:31:02.556770    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ceeffa5_00cc_4a12_8da7_fd463083e22a.slice/crio-72675ee5131d7435901813a8fa5588c8a9e8a80b0c13e43bd667cdededf79b27 WatchSource:0}: Error finding container 72675ee5131d7435901813a8fa5588c8a9e8a80b0c13e43bd667cdededf79b27: Status 404 returned error can't find the container with id 72675ee5131d7435901813a8fa5588c8a9e8a80b0c13e43bd667cdededf79b27
Nov 03 15:31:02.559990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:02.559936    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:02.560102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:02.560028    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:02.560102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:02.560061    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:03.494133 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:03.494084    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" event={"ID":"0ceeffa5-00cc-4a12-8da7-fd463083e22a","Type":"ContainerStarted","Data":"4d0e4fec4598de0bdfbeb48aa45ef884f50f179238353f2dd5855b9917bcafe9"}
Nov 03 15:31:03.494345 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:03.494154    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" event={"ID":"0ceeffa5-00cc-4a12-8da7-fd463083e22a","Type":"ContainerStarted","Data":"72675ee5131d7435901813a8fa5588c8a9e8a80b0c13e43bd667cdededf79b27"}
Nov 03 15:31:03.669267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:03.669210    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podStartSLOduration=2.66918977 podStartE2EDuration="2.66918977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:03.617798819 +0000 UTC m=+1378.542616732" watchObservedRunningTime="2026-11-03 15:31:03.66918977 +0000 UTC m=+1378.594007682"
Nov 03 15:31:04.116658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:04.116622    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:04.118161 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:04.118133    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 15:31:04.118295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:04.118190    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 15:31:05.135998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:05.131396    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 15:31:05.135998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:05.131476    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 15:31:06.116306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:06.116274    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 15:31:06.116464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:06.116330    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 15:31:07.116335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.116298    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 15:31:07.116834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.116371    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 15:31:07.615330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.615297    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"]
Nov 03 15:31:07.621415 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.621389    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:07.773006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.750469    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-77d0e91e-openshift-default-dockercfg-tbgg7\""
Nov 03 15:31:07.996707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.945558    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"]
Nov 03 15:31:07.997032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.979412    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.018994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:07.997218    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054492    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054571    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtf9\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054666    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054705    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054771    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054840    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.058996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.054942    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.118082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.117718    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 15:31:08.118082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.117786    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163038    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163102    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163203    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163271    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtf9\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163400    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163438    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163497    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163571    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.163682    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.164210    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.164829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.164533    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.213401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.168654    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.213401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.169299    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.213401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.170028    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.259088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.252992    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.259088 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.253807    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.274583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.274516    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtf9\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.277728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.277617    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token\") pod \"e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") " pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.556713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.556616    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:08.997915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:08.997884    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"]
Nov 03 15:31:08.999269 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:31:08.999243    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cee55e_2573_4c85_9ca4_b50a3a5e669a.slice/crio-e5abda46e0fd5962cad52ca8b3eff163b4277ef0a98e4585c223182f18fb3204 WatchSource:0}: Error finding container e5abda46e0fd5962cad52ca8b3eff163b4277ef0a98e4585c223182f18fb3204: Status 404 returned error can't find the container with id e5abda46e0fd5962cad52ca8b3eff163b4277ef0a98e4585c223182f18fb3204
Nov 03 15:31:09.001835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.001799    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:09.001931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.001865    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:09.001931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.001896    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:09.116761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.116722    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 15:31:09.116939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.116787    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 15:31:09.568682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.568641    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" event={"ID":"50cee55e-2573-4c85-9ca4-b50a3a5e669a","Type":"ContainerStarted","Data":"5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"}
Nov 03 15:31:09.569140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.568701    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" event={"ID":"50cee55e-2573-4c85-9ca4-b50a3a5e669a","Type":"ContainerStarted","Data":"e5abda46e0fd5962cad52ca8b3eff163b4277ef0a98e4585c223182f18fb3204"}
Nov 03 15:31:09.796746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.796591    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podStartSLOduration=2.796569483 podStartE2EDuration="2.796569483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:09.743210699 +0000 UTC m=+1384.668028613" watchObservedRunningTime="2026-11-03 15:31:09.796569483 +0000 UTC m=+1384.721387397"
Nov 03 15:31:09.846297 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.846188    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"]
Nov 03 15:31:09.851553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.851529    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:09.907869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.907834    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-47b989b3-serving-cert\""
Nov 03 15:31:09.960995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:09.951524    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"]
Nov 03 15:31:10.014660 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.014551    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbvh\" (UniqueName: \"kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.015124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.015101    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.015464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.015433    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.015550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.015517    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.133044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.132115    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.133044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.132296    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.133044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.132350    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.133044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.132446    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbvh\" (UniqueName: \"kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.133610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.133583    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.226818 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.226785    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.239777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.239718    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbvh\" (UniqueName: \"kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.255006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.251882    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls\") pod \"maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.263710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.263346    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:10.579097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.561332    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:10.672121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.672037    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:10.753166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.753074    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:10.908303 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.908272    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"]
Nov 03 15:31:10.908631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:10.908614    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:31:11.293113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:11.291163    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:31:11.293113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:11.291224    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:11.544059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:11.543527    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:11.798067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:11.795690    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l" event={"ID":"c97e1243-5959-4c4d-8a0b-89796d655665","Type":"ContainerStarted","Data":"6b0fa692375125b9119c1860cc25949a4899240b06ab494ff3d70bbe382cb7c8"}
Nov 03 15:31:11.965126 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:11.965077    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.041227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.041179    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.176254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.175236    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:31:12.228077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.227428    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.228077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.227530    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:31:12.353026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.348338    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:31:12.373452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.369886    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.373452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.370865    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.373452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.372293    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:31:12.373452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.372358    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.490028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.476663    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:12.490028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.480410    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.840997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.836937    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:12.959312 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:12.959211    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:13.022090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.017146    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:31:13.076008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.047931    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:13.076008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.048686    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:31:13.076008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.051153    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:13.082059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.082026    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:31:13.168304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.168257    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:13.204027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.178116    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:31:13.204027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.178689    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:31:13.237373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.237246    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:13.505417 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.487566    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:31:13.546997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.546110    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:31:13.575479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.563989    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:31:13.622037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.583160    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:31:13.622037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.583239    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:31:13.622037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:13.592553    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:31:14.069248 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.069208    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:14.099098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.069206    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:31:14.204308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.204263    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:31:14.224774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.224726    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:31:14.263667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.263451    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:31:14.316017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.267056    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:31:14.321082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.320936    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:14.558998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.557739    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:31:14.586220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.586113    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:14.856053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.802020    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Nov 03 15:31:14.856053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.822042    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:14.856053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.822478    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:31:14.948496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:14.948452    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.106915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.095658    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.106915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.103622    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:31:15.258920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.258881    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.305167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.305123    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.394086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.320087    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.394086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.321442    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.449733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.449690    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:31:15.450033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.450002    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.626997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.605376    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.627702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.627644    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:15.710051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.691502    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:31:15.847061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:15.847021    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:16.311869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.311820    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:31:16.313687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.313657    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:31:16.501457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.501402    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:31:16.964778 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.952399    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:16.964778 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.954059    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:16.987530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.983835    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:16.987530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.986147    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:16.987530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.986961    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:31:16.988150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:16.987895    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:31:17.335755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.335648    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:31:17.349029 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.338008    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:31:17.441388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.441352    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:31:17.441730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.441698    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:17.552637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.552589    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:17.566217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.557682    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" podUID="a4848f69-a0a9-4007-9203-7c656b86ef62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 03 15:31:17.575236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.557804    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Nov 03 15:31:17.895458 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895414    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l" event={"ID":"c97e1243-5959-4c4d-8a0b-89796d655665","Type":"ContainerStarted","Data":"a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078"}
Nov 03 15:31:17.895656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895493    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:17.895656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895546    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:31:17.895656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895575    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:31:17.895656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895607    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:31:17.895656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895637    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:31:17.895907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.895691    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:31:17.906705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.906676    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:31:17.924552 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.924503    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:31:17.946491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.938587    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:31:17.946491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:17.943162    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:31:18.116196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:18.116136    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l" podStartSLOduration=7.857057856 podStartE2EDuration="9.116120405s" totalImagesPullingTime="1.259062549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:17.993085431 +0000 UTC m=+1392.917903336" watchObservedRunningTime="2026-11-03 15:31:18.116120405 +0000 UTC m=+1393.040938317"
Nov 03 15:31:21.925013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:21.924958    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"]
Nov 03 15:31:21.929120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:21.929102    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:21.982916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:21.982886    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d785c53b-serving-cert\""
Nov 03 15:31:22.024019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.023671    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.024019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.023820    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.024261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.024058    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.024261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.024174    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc69\" (UniqueName: \"kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.039286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.039249    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"]
Nov 03 15:31:22.150045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.144810    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.150507 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.150476    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.150765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.150750    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc69\" (UniqueName: \"kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.159030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.151019    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.159346 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.152514    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.202913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.202727    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.209884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.209849    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.263337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.263307    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc69\" (UniqueName: \"kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69\") pod \"maas-api-e2e-derive-d785c53b-744f557747-sccqh\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.543188 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.543102    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:22.931962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:22.931937    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"]
Nov 03 15:31:22.932591 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:31:22.932570    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3925b57b_a182_4168_b45f_5fb70ffe2e00.slice/crio-9acff39ba2cc829a19c18543c431ace72cdcb865d851fa8b567082ddda327343 WatchSource:0}: Error finding container 9acff39ba2cc829a19c18543c431ace72cdcb865d851fa8b567082ddda327343: Status 404 returned error can't find the container with id 9acff39ba2cc829a19c18543c431ace72cdcb865d851fa8b567082ddda327343
Nov 03 15:31:23.752860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:23.752830    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:31:23.785453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:23.785415    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh" event={"ID":"3925b57b-a182-4168-b45f-5fb70ffe2e00","Type":"ContainerStarted","Data":"9acff39ba2cc829a19c18543c431ace72cdcb865d851fa8b567082ddda327343"}
Nov 03 15:31:24.794370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:24.794336    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh" event={"ID":"3925b57b-a182-4168-b45f-5fb70ffe2e00","Type":"ContainerStarted","Data":"c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b"}
Nov 03 15:31:24.794831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:24.794450    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:24.929562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:24.929505    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh" podStartSLOduration=3.10964859 podStartE2EDuration="3.929491527s" totalImagesPullingTime="819.842937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:24.926021895 +0000 UTC m=+1399.850839809" watchObservedRunningTime="2026-11-03 15:31:24.929491527 +0000 UTC m=+1399.854309441"
Nov 03 15:31:27.940126 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:31:27.940086    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:31:29.718306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.718270    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"]
Nov 03 15:31:29.722579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.722561    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.769055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.769022    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"]
Nov 03 15:31:29.773374 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.773351    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.810774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.810748    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n82lg\""
Nov 03 15:31:29.827299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827270    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5727af-ba07-43b8-a640-da62782a09c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827392    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xntqp\" (UniqueName: \"kubernetes.io/projected/9c5727af-ba07-43b8-a640-da62782a09c5-kube-api-access-xntqp\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827440    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827460    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5727af-ba07-43b8-a640-da62782a09c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827605    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5727af-ba07-43b8-a640-da62782a09c5-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827648    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827736 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827709    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827825 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827739    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827825 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827761    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.827983 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.827930    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"]
Nov 03 15:31:29.878232 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.878195    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"]
Nov 03 15:31:29.929257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929221    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.929438 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929403    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5727af-ba07-43b8-a640-da62782a09c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.929491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929446    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.929491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929473    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.929632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929612    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xntqp\" (UniqueName: \"kubernetes.io/projected/9c5727af-ba07-43b8-a640-da62782a09c5-kube-api-access-xntqp\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.929673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929653    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.929727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929679    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.929771 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929750    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.929815 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929778    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5727af-ba07-43b8-a640-da62782a09c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.929854 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929840    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5727af-ba07-43b8-a640-da62782a09c5-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.929897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929871    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.929941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929896    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmczh\" (UniqueName: \"kubernetes.io/projected/92fdb587-5d2d-4d2e-af75-d69fade7bd20-kube-api-access-qmczh\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.930037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.929927    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.930103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930042    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930070    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.930215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930128    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930187    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:29.930215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930203    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930214    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930263    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5727af-ba07-43b8-a640-da62782a09c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930364    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930408    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.930613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.930591    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.932514 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.932494    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5727af-ba07-43b8-a640-da62782a09c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.932650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.932632    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5727af-ba07-43b8-a640-da62782a09c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.992436 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.992346    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xntqp\" (UniqueName: \"kubernetes.io/projected/9c5727af-ba07-43b8-a640-da62782a09c5-kube-api-access-xntqp\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:29.993814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:29.993783    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5727af-ba07-43b8-a640-da62782a09c5-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-krrc2\" (UID: \"9c5727af-ba07-43b8-a640-da62782a09c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:30.031959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.031924    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.031998    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032048    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032062    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032155    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmczh\" (UniqueName: \"kubernetes.io/projected/92fdb587-5d2d-4d2e-af75-d69fade7bd20-kube-api-access-qmczh\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032182    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032214    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032250    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032318    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032326    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032382    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032545    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.032705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.032634    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.033069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.033027    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.034522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.034503    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.034641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.034624    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.036684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.036667    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:30.096442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.096417    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmczh\" (UniqueName: \"kubernetes.io/projected/92fdb587-5d2d-4d2e-af75-d69fade7bd20-kube-api-access-qmczh\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.096645 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.096583    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92fdb587-5d2d-4d2e-af75-d69fade7bd20-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-tq5bn\" (UID: \"92fdb587-5d2d-4d2e-af75-d69fade7bd20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.387129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.387037    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:30.471060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.470102    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"]
Nov 03 15:31:30.475907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.475851    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:30.476133 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.476108    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:30.476217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.476163    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:30.794339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.794313    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"]
Nov 03 15:31:30.797157 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:31:30.797126    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fdb587_5d2d_4d2e_af75_d69fade7bd20.slice/crio-6fdc1f09bdcf19c55c31bdc541d0f2dbe350c2e95130813af51d3f169a6f9042 WatchSource:0}: Error finding container 6fdc1f09bdcf19c55c31bdc541d0f2dbe350c2e95130813af51d3f169a6f9042: Status 404 returned error can't find the container with id 6fdc1f09bdcf19c55c31bdc541d0f2dbe350c2e95130813af51d3f169a6f9042
Nov 03 15:31:30.799946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.799913    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:30.800042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.799998    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:30.800042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.800031    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:30.809680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.809655    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:31:30.853091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.853059    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" event={"ID":"9c5727af-ba07-43b8-a640-da62782a09c5","Type":"ContainerStarted","Data":"e7012c1567766dc544dc06b33d290f1c50f91f7309df8738bc1f6ce05597fff4"}
Nov 03 15:31:30.853253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.853123    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" event={"ID":"9c5727af-ba07-43b8-a640-da62782a09c5","Type":"ContainerStarted","Data":"a3cf012bcfd555ea2f5da6f65888078be23c81b5d79a070d7d8ac80c1bca3153"}
Nov 03 15:31:30.854750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:30.854725    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" event={"ID":"92fdb587-5d2d-4d2e-af75-d69fade7bd20","Type":"ContainerStarted","Data":"6fdc1f09bdcf19c55c31bdc541d0f2dbe350c2e95130813af51d3f169a6f9042"}
Nov 03 15:31:31.048713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:31.046370    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:31.058961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:31.058920    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:31.059210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:31.059017    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:31.061079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:31.061035    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podStartSLOduration=2.061017783 podStartE2EDuration="2.061017783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:31.060663342 +0000 UTC m=+1405.985481256" watchObservedRunningTime="2026-11-03 15:31:31.061017783 +0000 UTC m=+1405.985835700"
Nov 03 15:31:31.891249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:31.891214    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" event={"ID":"92fdb587-5d2d-4d2e-af75-d69fade7bd20","Type":"ContainerStarted","Data":"8d29a8d49d41ee212983433df183ceba97c2fd45785eaec75b3cd8b9be75b282"}
Nov 03 15:31:32.020674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:32.020595    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podStartSLOduration=3.020580142 podStartE2EDuration="3.020580142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:32.017539403 +0000 UTC m=+1406.942357313" watchObservedRunningTime="2026-11-03 15:31:32.020580142 +0000 UTC m=+1406.945398055"
Nov 03 15:31:32.069995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:32.054415    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:32.069995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:32.054482    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:32.413998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:32.391386    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:32.413998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:32.401580    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:32.413998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:32.401638    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:33.049997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:33.048389    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:33.049997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:33.048453    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:33.388046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:33.387920    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:33.388046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:33.388005    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:34.037995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:34.037412    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:34.037995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:34.037475    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:34.395059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:34.391590    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:34.395059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:34.391644    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:35.040997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:35.037452    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:35.040997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:35.037518    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:35.391057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:35.390479    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:35.391057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:35.390539    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:36.044082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:36.040441    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:36.044082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:36.040509    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:36.387937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:36.387848    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:36.387937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:36.387913    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:37.039996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:37.037384    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:37.039996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:37.037456    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:37.387831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:37.387724    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:37.387831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:37.387807    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:38.037484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:38.037445    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 15:31:38.037667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:38.037506    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 15:31:38.387939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:38.387855    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 15:31:38.387939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:38.387921    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 15:31:39.568653 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:39.568617    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:39.569435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:39.569415    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:39.570434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:39.570410    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2"
Nov 03 15:31:39.787403 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:39.787371    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:39.787704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:39.787682    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:39.788867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:39.788834    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn"
Nov 03 15:31:44.840733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.840698    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"]
Nov 03 15:31:44.846329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.846304    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.891395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.891366    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"]
Nov 03 15:31:44.895574 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.895550    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:44.946127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946087    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcs7\" (UniqueName: \"kubernetes.io/projected/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-kube-api-access-4pcs7\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946147    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946283    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946310    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946342    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946451    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946518    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.946597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.946576    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.947114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.947092    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:44.950824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:44.950799    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"]
Nov 03 15:31:45.004889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.004851    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"]
Nov 03 15:31:45.048627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048591    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.048794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048688    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcs7\" (UniqueName: \"kubernetes.io/projected/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-kube-api-access-4pcs7\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.048794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048741    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.048794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048788    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.048932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048887    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.048932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048913    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048932    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.049068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.048950    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049070    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049097    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.049257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049178    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049240    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049288    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.049373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049327    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.049481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049371    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049406    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b46a7-66e5-4991-ae83-a6086edb8cfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.049481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049437    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049498    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049575    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmxrx\" (UniqueName: \"kubernetes.io/projected/932b46a7-66e5-4991-ae83-a6086edb8cfe-kube-api-access-lmxrx\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.049689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049621    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049632    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.049785 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.049705    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.050227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.050210    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.052249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.052225    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.052502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.052483    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.111056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.110922    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcs7\" (UniqueName: \"kubernetes.io/projected/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-kube-api-access-4pcs7\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.111056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.111011    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932ea7e5-0b88-4fbf-a11a-1e8a34e6b677-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-7w5hv\" (UID: \"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.151553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151521    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmxrx\" (UniqueName: \"kubernetes.io/projected/932b46a7-66e5-4991-ae83-a6086edb8cfe-kube-api-access-lmxrx\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151575    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151619    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151654    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151696    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151747    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151830    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.151904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151859    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.152080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.151905    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b46a7-66e5-4991-ae83-a6086edb8cfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.152171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.152137    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.152234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.152195    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.152357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.152334    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.152596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.152575    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.152643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.152607    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b46a7-66e5-4991-ae83-a6086edb8cfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.154124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.154103    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.154376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.154355    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.161267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.161247    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:45.220295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.220263    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmxrx\" (UniqueName: \"kubernetes.io/projected/932b46a7-66e5-4991-ae83-a6086edb8cfe-kube-api-access-lmxrx\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.221312 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.221241    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b46a7-66e5-4991-ae83-a6086edb8cfe-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-5zkfh\" (UID: \"932b46a7-66e5-4991-ae83-a6086edb8cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.510499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.510469    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:45.600461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.600432    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"]
Nov 03 15:31:45.606630 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:31:45.606596    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932ea7e5_0b88_4fbf_a11a_1e8a34e6b677.slice/crio-74347a535217fe548018213bd50d39ee62350e682c50b5475bbf50bd04898245 WatchSource:0}: Error finding container 74347a535217fe548018213bd50d39ee62350e682c50b5475bbf50bd04898245: Status 404 returned error can't find the container with id 74347a535217fe548018213bd50d39ee62350e682c50b5475bbf50bd04898245
Nov 03 15:31:45.610128 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.610079    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:45.610238 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.610182    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:45.610238 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.610227    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:45.901384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.901353    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"]
Nov 03 15:31:45.901810 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:31:45.901632    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932b46a7_66e5_4991_ae83_a6086edb8cfe.slice/crio-1afe464a987a8e7d77967a470ebe0a0e631dea8718f48225f52f1149b1c78aac WatchSource:0}: Error finding container 1afe464a987a8e7d77967a470ebe0a0e631dea8718f48225f52f1149b1c78aac: Status 404 returned error can't find the container with id 1afe464a987a8e7d77967a470ebe0a0e631dea8718f48225f52f1149b1c78aac
Nov 03 15:31:45.904643 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.904607    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:45.904759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.904672    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:45.904759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:45.904699    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:31:46.081155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.081117    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" event={"ID":"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677","Type":"ContainerStarted","Data":"2f91cf1ba9c3fb92c286b62cea9c95f1e5c9df93828a373beb00df0b25a435fd"}
Nov 03 15:31:46.081304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.081186    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" event={"ID":"932ea7e5-0b88-4fbf-a11a-1e8a34e6b677","Type":"ContainerStarted","Data":"74347a535217fe548018213bd50d39ee62350e682c50b5475bbf50bd04898245"}
Nov 03 15:31:46.083071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.083045    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" event={"ID":"932b46a7-66e5-4991-ae83-a6086edb8cfe","Type":"ContainerStarted","Data":"4ad2f3727b5b025408d0f86c0202debd3481bd88c371e485a1ff7850eff2288d"}
Nov 03 15:31:46.083202 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.083104    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" event={"ID":"932b46a7-66e5-4991-ae83-a6086edb8cfe","Type":"ContainerStarted","Data":"1afe464a987a8e7d77967a470ebe0a0e631dea8718f48225f52f1149b1c78aac"}
Nov 03 15:31:46.161949 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.161858    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:46.163184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.163154    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:46.163303 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.163222    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:46.207808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.207743    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podStartSLOduration=2.20772504 podStartE2EDuration="2.20772504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:46.203737613 +0000 UTC m=+1421.128555526" watchObservedRunningTime="2026-11-03 15:31:46.20772504 +0000 UTC m=+1421.132542956"
Nov 03 15:31:46.327119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.327063    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podStartSLOduration=2.327046888 podStartE2EDuration="2.327046888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:31:46.326263788 +0000 UTC m=+1421.251081695" watchObservedRunningTime="2026-11-03 15:31:46.327046888 +0000 UTC m=+1421.251864800"
Nov 03 15:31:46.510784 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.510681    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:46.512814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.512784    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:46.512951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:46.512844    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:47.162600 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:47.162567    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:47.163124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:47.162646    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:47.511022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:47.510924    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:47.511022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:47.510986    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:48.173994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:48.170118    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:48.173994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:48.170195    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:48.510938 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:48.510851    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:48.510938 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:48.510916    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:49.184998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:49.184939    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:49.185566 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:49.185536    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:49.511520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:49.511427    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:49.511520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:49.511486    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:50.168406 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:50.168367    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:50.168713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:50.168681    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:50.511576 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:50.511489    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:50.511576 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:50.511546    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:51.171995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:51.169233    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:51.171995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:51.169290    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:51.511462 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:51.511378    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:51.511462 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:51.511437    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:52.162576 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:52.162531    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:52.162793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:52.162595    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:52.511397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:52.511319    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:52.511397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:52.511384    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:53.171993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:53.168236    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 15:31:53.171993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:53.168305    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 15:31:53.511398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:53.511310    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:53.511398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:53.511376    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:54.511589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:54.511553    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 15:31:54.512001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:54.511630    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 15:31:54.679525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:54.679492    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:54.680384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:54.680356    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:54.681210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:54.681182    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv"
Nov 03 15:31:55.575000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:55.572255    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:31:55.798653 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:55.798617    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:55.799187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:55.799162    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:55.800093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:55.800073    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh"
Nov 03 15:31:58.200219 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:31:58.200178    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:31:58.852131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:58.852076    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:59.108550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:59.107475    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:31:59.305998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:59.261130    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:31:59.335898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:59.335843    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:31:59.377054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:59.372098    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:31:59.402999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:59.399311    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:31:59.847013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:31:59.846644    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:00.030998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.030074    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:00.293410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.143173    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:00.293410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.198207    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:32:00.326839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.324358    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:32:00.327367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.326550    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:32:00.474007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.473015    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:32:00.536891 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.500433    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:00.536891 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.507147    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:00.587950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.573663    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:32:00.587950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.573752    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:32:00.587950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.573806    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:32:00.587950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.573843    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:32:00.661211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.601486    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6" podUID="cff4535e-af16-4971-8c47-6777f461d121" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:00.661211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.602101    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:32:00.661211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.602347    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:32:00.899331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:00.899290    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:01.086042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.084472    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.116170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.116120    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:32:01.154659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.153205    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.197300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.197268    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:32:01.197594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.197577    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:32:01.375876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.375832    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:01.403125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.403058    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:01.411889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.411770    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Nov 03 15:32:01.557346 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.557305    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Nov 03 15:32:01.605290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.605250    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4xf6" podUID="a4848f69-a0a9-4007-9203-7c656b86ef62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.632121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.632081    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:01.636993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.632636    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.636993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.633994    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Nov 03 15:32:01.658311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.652328    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.678190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.678084    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.678858 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.678818    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:32:01.760425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.760384    4955 patch_prober.go:29] interesting pod/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:32:01.760627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.760450    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:01.923560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:01.923509    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:32:02.050098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.038432    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:32:02.095208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.095166    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.154479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.154413    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.324626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.324515    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 03 15:32:02.475037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.474987    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.553024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.551663    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.633944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.633843    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.653102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.653065    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:32:02.654024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.653562    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.683170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.683103    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.685802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.685737    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:32:02.721159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.721098    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:02.933997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:02.925197    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:03.129075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.127262    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:32:03.192056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.179922    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:32:03.542056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.506177    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:03.744001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.741895    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:32:03.744001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.742238    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:32:03.765523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.765480    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:03.768102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.768059    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:32:03.811698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:03.811602    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.107737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.107651    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:32:04.109215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.109195    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:32:04.114389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.114063    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.185997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.160052    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:32:04.230915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.229166    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.778180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.778141    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:32:04.807211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.788659    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.808457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.791848    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:32:04.809683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.809637    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.814630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.814593    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.815130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.814722    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:04.901793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.901640    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:32:04.910423 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.910393    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:32:04.911303 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.911276    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:32:04.920251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.920227    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:32:04.923877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.923736    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:32:04.924008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:04.923939    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:32:10.757132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:10.757089    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:32:10.757566 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:10.757161    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:12.452031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.451988    4955 generic.go:356] "Generic (PLEG): container finished" podID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerID="78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14" exitCode=0
Nov 03 15:32:12.452505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.452064    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" event={"ID":"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde","Type":"ContainerDied","Data":"78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"}
Nov 03 15:32:12.452610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.452593    4955 scope.go:122] "RemoveContainer" containerID="78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"
Nov 03 15:32:12.453875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.453857    4955 generic.go:356] "Generic (PLEG): container finished" podID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerID="f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f" exitCode=0
Nov 03 15:32:12.453961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.453917    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" event={"ID":"d6548c2a-1484-4966-a9b9-abb8c70ae652","Type":"ContainerDied","Data":"f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"}
Nov 03 15:32:12.454339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.454321    4955 scope.go:122] "RemoveContainer" containerID="f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"
Nov 03 15:32:12.454468 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.454439    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:12.454525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.454510    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:12.454575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.454542    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:12.455870 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.455841    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:12.455963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.455900    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:12.455963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:12.455931    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:13.485683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.485647    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" event={"ID":"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde","Type":"ContainerStarted","Data":"6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237"}
Nov 03 15:32:13.488357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.488332    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" event={"ID":"d6548c2a-1484-4966-a9b9-abb8c70ae652","Type":"ContainerStarted","Data":"8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7"}
Nov 03 15:32:13.823113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.822915    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:32:13.823321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.823298    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 15:32:13.823498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.823358    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 15:32:13.928434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.928400    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:32:13.928630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.928553    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:32:13.928630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:13.928599    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:32:14.821987 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:14.821928    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 15:32:14.822417 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:14.822015    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 15:32:14.952994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:14.952308    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:32:14.952994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:14.952366    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:32:15.822664 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:15.822632    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 15:32:15.823070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:15.822683    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 15:32:15.948995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:15.928960    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:32:15.948995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:15.929061    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:32:16.822233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:16.822191    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 15:32:16.822450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:16.822273    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 15:32:16.928900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:16.928865    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:32:16.929451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:16.928922    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:32:17.822471 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:17.822436    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 15:32:17.822694 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:17.822495    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 15:32:17.945996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:17.928409    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:32:17.945996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:17.928459    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:32:18.645803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:18.644608    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:32:18.824402 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:18.824364    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 15:32:18.824685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:18.824648    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 15:32:18.930196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:18.930109    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 15:32:18.930495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:18.930460    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 15:32:19.827358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:19.827326    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:32:19.828514 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:19.828489    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:32:19.932467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:19.932434    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:32:19.932723 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:19.932704    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:32:19.933349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:19.933326    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:32:27.939765 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:32:27.939720    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:32:33.155112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:33.155078    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:32:41.791123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:41.791084    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:32:41.791563 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:41.791137    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:32:45.436710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:45.436668    4955 generic.go:356] "Generic (PLEG): container finished" podID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerID="5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79" exitCode=0
Nov 03 15:32:45.437516 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:45.437408    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" event={"ID":"50cee55e-2573-4c85-9ca4-b50a3a5e669a","Type":"ContainerDied","Data":"5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"}
Nov 03 15:32:45.449013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:45.438201    4955 scope.go:122] "RemoveContainer" containerID="5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"
Nov 03 15:32:45.453064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:45.453015    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:45.453701 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:45.453654    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:45.453825 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:45.453739    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:32:46.463824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:46.463792    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" event={"ID":"50cee55e-2573-4c85-9ca4-b50a3a5e669a","Type":"ContainerStarted","Data":"78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4"}
Nov 03 15:32:46.558250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:46.558191    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:32:46.558502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:46.558474    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 15:32:46.558567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:46.558547    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 15:32:47.557995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:47.557414    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 15:32:47.557995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:47.557478    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 15:32:48.557871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:48.557835    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 15:32:48.558158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:48.557905    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 15:32:49.558041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:49.558001    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 15:32:49.558219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:49.558077    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 15:32:50.557854 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:50.557819    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 15:32:50.558094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:50.557896    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 15:32:51.558327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:51.558286    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 15:32:51.558824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:51.558384    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 15:32:52.562168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:52.562133    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:32:52.562668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:52.562627    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:32:52.563377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:32:52.563356    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:32:57.954842 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:32:57.954801    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:33:05.651924 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:05.651891    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:33:05.657517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:05.657487    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:33:05.665492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:05.665465    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:33:05.675481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:05.675455    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:33:07.639006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:07.638203    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:33:07.689106 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:07.689055    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:33:07.690138 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:07.690044    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:33:14.553077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:14.547174    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"]
Nov 03 15:33:14.583025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:14.566051    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l" podUID="c97e1243-5959-4c4d-8a0b-89796d655665" containerName="maas-api" containerID="cri-o://a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078" gracePeriod=30
Nov 03 15:33:15.319192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.319155    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:33:15.476995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.472048    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs\") pod \"c97e1243-5959-4c4d-8a0b-89796d655665\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") "
Nov 03 15:33:15.476995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.472175    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh\" (UniqueName: \"kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh\") pod \"c97e1243-5959-4c4d-8a0b-89796d655665\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") "
Nov 03 15:33:15.476995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.472338    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls\") pod \"c97e1243-5959-4c4d-8a0b-89796d655665\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") "
Nov 03 15:33:15.476995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.472493    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca\") pod \"c97e1243-5959-4c4d-8a0b-89796d655665\" (UID: \"c97e1243-5959-4c4d-8a0b-89796d655665\") "
Nov 03 15:33:15.476995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.475159    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca" pod "c97e1243-5959-4c4d-8a0b-89796d655665" (UID: "c97e1243-5959-4c4d-8a0b-89796d655665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:33:15.531015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.524145    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls" pod "c97e1243-5959-4c4d-8a0b-89796d655665" (UID: "c97e1243-5959-4c4d-8a0b-89796d655665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:33:15.531015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.524649    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh" pod "c97e1243-5959-4c4d-8a0b-89796d655665" (UID: "c97e1243-5959-4c4d-8a0b-89796d655665"). InnerVolumeSpecName "kube-api-access-ksbvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:33:15.629049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.628664    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:15.629049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.628702    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e1243-5959-4c4d-8a0b-89796d655665-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:15.629049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.628717    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksbvh\" (UniqueName: \"kubernetes.io/projected/c97e1243-5959-4c4d-8a0b-89796d655665-kube-api-access-ksbvh\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:15.721874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.721805    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs" pod "c97e1243-5959-4c4d-8a0b-89796d655665" (UID: "c97e1243-5959-4c4d-8a0b-89796d655665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:33:15.732371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.732332    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c97e1243-5959-4c4d-8a0b-89796d655665-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:15.876498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.876464    4955 generic.go:356] "Generic (PLEG): container finished" podID="c97e1243-5959-4c4d-8a0b-89796d655665" containerID="a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078" exitCode=0
Nov 03 15:33:15.876668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.876521    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"
Nov 03 15:33:15.876668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.876547    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l" event={"ID":"c97e1243-5959-4c4d-8a0b-89796d655665","Type":"ContainerDied","Data":"a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078"}
Nov 03 15:33:15.876668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.876610    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l" event={"ID":"c97e1243-5959-4c4d-8a0b-89796d655665","Type":"ContainerDied","Data":"6b0fa692375125b9119c1860cc25949a4899240b06ab494ff3d70bbe382cb7c8"}
Nov 03 15:33:15.876668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.876639    4955 scope.go:122] "RemoveContainer" containerID="a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078"
Nov 03 15:33:15.887939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.887919    4955 scope.go:122] "RemoveContainer" containerID="a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078"
Nov 03 15:33:15.888231 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:15.888212    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078\": container with ID starting with a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078 not found: ID does not exist" containerID="a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078"
Nov 03 15:33:15.888286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:15.888240    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078"} err="failed to get container status \"a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078\": rpc error: code = NotFound desc = could not find container \"a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078\": container with ID starting with a18ce5eb66cb815ab5d04f46ccb8c777521d3ac10b311bb375a39b8603b7f078 not found: ID does not exist"
Nov 03 15:33:16.045889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:16.045859    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"]
Nov 03 15:33:16.049335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:16.049312    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-47b989b3-8fcf8ff7-gt65l"]
Nov 03 15:33:17.141069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:17.141030    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"]
Nov 03 15:33:17.141874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:17.141841    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" containerID="cri-o://8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7" gracePeriod=30
Nov 03 15:33:17.207766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:17.207733    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97e1243-5959-4c4d-8a0b-89796d655665" path="/var/lib/kubelet/pods/c97e1243-5959-4c4d-8a0b-89796d655665/volumes"
Nov 03 15:33:19.272066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.272022    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"]
Nov 03 15:33:19.272963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.272943    4955 cpu_manager.go:414] "containerMap: removing container" podUID="c97e1243-5959-4c4d-8a0b-89796d655665" containerName="maas-api"
Nov 03 15:33:19.272963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.272984    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97e1243-5959-4c4d-8a0b-89796d655665" containerName="maas-api"
Nov 03 15:33:19.273127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.273095    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97e1243-5959-4c4d-8a0b-89796d655665" containerName="maas-api"
Nov 03 15:33:19.278092 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.278068    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.331582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.331551    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-dfec2d6c-gw-openshift-default-dockercfg-t6wcz\""
Nov 03 15:33:19.362869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.362828    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.362913    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363036    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363098    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2v6c\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363251    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363405 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363380    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363438    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363660    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.363756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.363739    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.386214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.386183    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"]
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.506684    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.506742    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.506789    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.506867    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2v6c\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.507015    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.507171    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.507230    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.507497    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.507625    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.526466    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.526892    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.527716    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.528015    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.528603    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.529042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.528855    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.593457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.592326    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.662857 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.662795    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.666329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.666301    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2v6c\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c\") pod \"e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") " pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:19.830025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.829899    4955 patch_prober.go:29] interesting pod/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:33:19.830025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.829962    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:33:19.892001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:19.891934    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:20.326904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:20.326869    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"]
Nov 03 15:33:20.363106 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:33:20.363076    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc673af27_2d27_46a3_a386_0a29dbb07d72.slice/crio-9fbe9ec171768b2724eb9f09b6db7b32eebe7aa030b5a036ec4bdf4bed30c478 WatchSource:0}: Error finding container 9fbe9ec171768b2724eb9f09b6db7b32eebe7aa030b5a036ec4bdf4bed30c478: Status 404 returned error can't find the container with id 9fbe9ec171768b2724eb9f09b6db7b32eebe7aa030b5a036ec4bdf4bed30c478
Nov 03 15:33:20.366017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:20.365939    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:33:20.366128 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:20.366077    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:33:20.366195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:20.366130    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:33:20.984050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:20.984019    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"]
Nov 03 15:33:20.984275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:20.984252    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh" podUID="3925b57b-a182-4168-b45f-5fb70ffe2e00" containerName="maas-api" containerID="cri-o://c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b" gracePeriod=30
Nov 03 15:33:21.000313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.000277    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" event={"ID":"c673af27-2d27-46a3-a386-0a29dbb07d72","Type":"ContainerStarted","Data":"084ab27cfed9de62e59da6bb95329235bdddc4fb606069855d5d264dffd6de77"}
Nov 03 15:33:21.000479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.000332    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" event={"ID":"c673af27-2d27-46a3-a386-0a29dbb07d72","Type":"ContainerStarted","Data":"9fbe9ec171768b2724eb9f09b6db7b32eebe7aa030b5a036ec4bdf4bed30c478"}
Nov 03 15:33:21.345334 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.342209    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podStartSLOduration=2.342189019 podStartE2EDuration="2.342189019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:33:21.34048196 +0000 UTC m=+1516.265299896" watchObservedRunningTime="2026-11-03 15:33:21.342189019 +0000 UTC m=+1516.267006934"
Nov 03 15:33:21.424864 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.424840    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:33:21.510995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.510502    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls\") pod \"3925b57b-a182-4168-b45f-5fb70ffe2e00\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") "
Nov 03 15:33:21.510995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.510575    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca\") pod \"3925b57b-a182-4168-b45f-5fb70ffe2e00\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") "
Nov 03 15:33:21.510995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.510757    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs\") pod \"3925b57b-a182-4168-b45f-5fb70ffe2e00\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") "
Nov 03 15:33:21.510995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.510830    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69\" (UniqueName: \"kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69\") pod \"3925b57b-a182-4168-b45f-5fb70ffe2e00\" (UID: \"3925b57b-a182-4168-b45f-5fb70ffe2e00\") "
Nov 03 15:33:21.534010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.530126    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69" pod "3925b57b-a182-4168-b45f-5fb70ffe2e00" (UID: "3925b57b-a182-4168-b45f-5fb70ffe2e00"). InnerVolumeSpecName "kube-api-access-8cc69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:33:21.557912 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.557872    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca" pod "3925b57b-a182-4168-b45f-5fb70ffe2e00" (UID: "3925b57b-a182-4168-b45f-5fb70ffe2e00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:33:21.574570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.571837    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls" pod "3925b57b-a182-4168-b45f-5fb70ffe2e00" (UID: "3925b57b-a182-4168-b45f-5fb70ffe2e00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:33:21.607069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.579156    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs" pod "3925b57b-a182-4168-b45f-5fb70ffe2e00" (UID: "3925b57b-a182-4168-b45f-5fb70ffe2e00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:33:21.657021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.654564    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:21.657021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.654603    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925b57b-a182-4168-b45f-5fb70ffe2e00-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:21.657021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.654619    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3925b57b-a182-4168-b45f-5fb70ffe2e00-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:21.657021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.654632    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc69\" (UniqueName: \"kubernetes.io/projected/3925b57b-a182-4168-b45f-5fb70ffe2e00-kube-api-access-8cc69\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:21.892235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.892205    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:21.897660 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:21.897633    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:22.072011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.066487    4955 generic.go:356] "Generic (PLEG): container finished" podID="3925b57b-a182-4168-b45f-5fb70ffe2e00" containerID="c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b" exitCode=0
Nov 03 15:33:22.072011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.067943    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"
Nov 03 15:33:22.082003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.074576    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh" event={"ID":"3925b57b-a182-4168-b45f-5fb70ffe2e00","Type":"ContainerDied","Data":"c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b"}
Nov 03 15:33:22.082003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.074658    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh" event={"ID":"3925b57b-a182-4168-b45f-5fb70ffe2e00","Type":"ContainerDied","Data":"9acff39ba2cc829a19c18543c431ace72cdcb865d851fa8b567082ddda327343"}
Nov 03 15:33:22.082003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.074692    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:22.082003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.074724    4955 scope.go:122] "RemoveContainer" containerID="c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b"
Nov 03 15:33:22.082003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.075123    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:33:22.271064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.270169    4955 scope.go:122] "RemoveContainer" containerID="c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b"
Nov 03 15:33:22.271064 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:22.270708    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b\": container with ID starting with c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b not found: ID does not exist" containerID="c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b"
Nov 03 15:33:22.271064 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.270747    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b"} err="failed to get container status \"c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b\": rpc error: code = NotFound desc = could not find container \"c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b\": container with ID starting with c209bee63ca7b2e9d1581fe88ba7f42d54a894d89f1868ce5056a2d281b4e14b not found: ID does not exist"
Nov 03 15:33:22.486009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.484911    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"]
Nov 03 15:33:22.490858 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.490821    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d785c53b-744f557747-sccqh"]
Nov 03 15:33:22.569560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.569527    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:33:22.605779 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.605743    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.605945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.605796    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.605945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.605907    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606053    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606114    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606140    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606183    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606194    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:22.606313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606218    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606284    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token\") pod \"d6548c2a-1484-4966-a9b9-abb8c70ae652\" (UID: \"d6548c2a-1484-4966-a9b9-abb8c70ae652\") "
Nov 03 15:33:22.606455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606434    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:22.606508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606477    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:22.606508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606485    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:33:22.606752 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606726    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:22.606987 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606945    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.606987 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606984    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.607119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.606998    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.607119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.607006    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6548c2a-1484-4966-a9b9-abb8c70ae652-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.608336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.608314    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:22.608426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.608371    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "kube-api-access-twj4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:33:22.608516 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.608501    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:33:22.609009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.608958    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo" pod "d6548c2a-1484-4966-a9b9-abb8c70ae652" (UID: "d6548c2a-1484-4966-a9b9-abb8c70ae652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:33:22.709213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.709126    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.709213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.709156    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.709213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.709167    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6548c2a-1484-4966-a9b9-abb8c70ae652-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.709213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.709175    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:22.709213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:22.709184    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twj4p\" (UniqueName: \"kubernetes.io/projected/d6548c2a-1484-4966-a9b9-abb8c70ae652-kube-api-access-twj4p\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:23.076352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.076265    4955 generic.go:356] "Generic (PLEG): container finished" podID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerID="8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7" exitCode=0
Nov 03 15:33:23.076352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.076314    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"
Nov 03 15:33:23.076592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.076350    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" event={"ID":"d6548c2a-1484-4966-a9b9-abb8c70ae652","Type":"ContainerDied","Data":"8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7"}
Nov 03 15:33:23.076592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.076435    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4" event={"ID":"d6548c2a-1484-4966-a9b9-abb8c70ae652","Type":"ContainerDied","Data":"806b95634ab00bd4bc204529c14100283fe9f81c85249bfb0efb6d1a4c85bacd"}
Nov 03 15:33:23.076592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.076466    4955 scope.go:122] "RemoveContainer" containerID="8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7"
Nov 03 15:33:23.087962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.087945    4955 scope.go:122] "RemoveContainer" containerID="f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"
Nov 03 15:33:23.097641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.097623    4955 scope.go:122] "RemoveContainer" containerID="8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7"
Nov 03 15:33:23.098013 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:23.097985    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7\": container with ID starting with 8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7 not found: ID does not exist" containerID="8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7"
Nov 03 15:33:23.098119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.098030    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7"} err="failed to get container status \"8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7\": rpc error: code = NotFound desc = could not find container \"8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7\": container with ID starting with 8487ac3fe31148499a66dc58fc63b40635010a68c621e4a93c61f0b0226e61a7 not found: ID does not exist"
Nov 03 15:33:23.098119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.098055    4955 scope.go:122] "RemoveContainer" containerID="f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"
Nov 03 15:33:23.098381 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:23.098360    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f\": container with ID starting with f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f not found: ID does not exist" containerID="f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"
Nov 03 15:33:23.098425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.098389    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f"} err="failed to get container status \"f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f\": rpc error: code = NotFound desc = could not find container \"f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f\": container with ID starting with f6b1b320403b08fc1313544d7480000480f7e95dc367268afd48d1d428264f0f not found: ID does not exist"
Nov 03 15:33:23.322993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.321957    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3925b57b-a182-4168-b45f-5fb70ffe2e00" path="/var/lib/kubelet/pods/3925b57b-a182-4168-b45f-5fb70ffe2e00/volumes"
Nov 03 15:33:23.337130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.334074    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"]
Nov 03 15:33:23.351000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:23.350510    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-47b989b3-openshift-default-7649594d7c-bwsd4"]
Nov 03 15:33:25.207622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:25.207591    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" path="/var/lib/kubelet/pods/d6548c2a-1484-4966-a9b9-abb8c70ae652/volumes"
Nov 03 15:33:26.973087 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.972960    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"]
Nov 03 15:33:26.975175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.975107    4955 cpu_manager.go:414] "containerMap: removing container" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy"
Nov 03 15:33:26.975347 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.975329    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy"
Nov 03 15:33:26.975487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.975473    4955 cpu_manager.go:414] "containerMap: removing container" podUID="3925b57b-a182-4168-b45f-5fb70ffe2e00" containerName="maas-api"
Nov 03 15:33:26.975623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.975607    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3925b57b-a182-4168-b45f-5fb70ffe2e00" containerName="maas-api"
Nov 03 15:33:26.975939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.975923    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy"
Nov 03 15:33:26.976173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.976141    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy"
Nov 03 15:33:26.976288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.976277    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="3925b57b-a182-4168-b45f-5fb70ffe2e00" containerName="maas-api"
Nov 03 15:33:26.991198 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:26.982358    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.064908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.064856    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.065805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.065774    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzdg\" (UniqueName: \"kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg\") pod \"payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.077560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.077521    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"]
Nov 03 15:33:27.089276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.089242    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-dfec2d6c\""
Nov 03 15:33:27.149296 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.149259    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-dfec2d6c-dockercfg-wrb5z\""
Nov 03 15:33:27.197066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.185118    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.307311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.307217    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzdg\" (UniqueName: \"kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg\") pod \"payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.324992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.311304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.477231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.477193    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"]
Nov 03 15:33:27.478634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.478604    4955 cpu_manager.go:414] "containerMap: removing container" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy"
Nov 03 15:33:27.478808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.478795    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6548c2a-1484-4966-a9b9-abb8c70ae652" containerName="istio-proxy"
Nov 03 15:33:27.502036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.499445    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzdg\" (UniqueName: \"kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg\") pod \"payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.505014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.503300    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.589755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.589711    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k55l\" (UniqueName: \"kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l\") pod \"payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.601009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.590150    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.601576 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.594839    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"]
Nov 03 15:33:27.666071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.666035    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:27.705015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.703366    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k55l\" (UniqueName: \"kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l\") pod \"payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.705015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.703448    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.705015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.704239    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.767645 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.767612    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k55l\" (UniqueName: \"kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l\") pod \"payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.905780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:27.905746    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:27.948067 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:27.946252    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:33:28.073995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:28.073716    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"]
Nov 03 15:33:28.263257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:28.263163    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr" event={"ID":"26ec121d-7419-4578-af73-45cb28d8c86f","Type":"ContainerStarted","Data":"bd19c23feb3c04cab7d5930858fd4d97356daf726e1375344e5297745ef7c269"}
Nov 03 15:33:28.327932 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:33:28.327903    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef82c8a_7949_45fd_a44e_6a5ed579c9fc.slice/crio-e11f73032b290fc15b9477cda9522fb061a70ecda15d74c611544713a1146746 WatchSource:0}: Error finding container e11f73032b290fc15b9477cda9522fb061a70ecda15d74c611544713a1146746: Status 404 returned error can't find the container with id e11f73032b290fc15b9477cda9522fb061a70ecda15d74c611544713a1146746
Nov 03 15:33:28.328398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:28.328374    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"]
Nov 03 15:33:29.271747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:29.271708    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf" event={"ID":"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc","Type":"ContainerStarted","Data":"e11f73032b290fc15b9477cda9522fb061a70ecda15d74c611544713a1146746"}
Nov 03 15:33:30.873995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:30.873282    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"]
Nov 03 15:33:30.873995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:30.873604    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" containerID="cri-o://6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237" gracePeriod=30
Nov 03 15:33:32.379738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:32.379698    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr" event={"ID":"26ec121d-7419-4578-af73-45cb28d8c86f","Type":"ContainerStarted","Data":"1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4"}
Nov 03 15:33:32.380221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:32.379767    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:32.381378 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:32.381345    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf" event={"ID":"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc","Type":"ContainerStarted","Data":"4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58"}
Nov 03 15:33:32.381533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:32.381409    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:32.511727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:32.511673    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr" podStartSLOduration=2.695433978 podStartE2EDuration="6.511658341s" totalImagesPullingTime="3.816224363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:33:32.509681752 +0000 UTC m=+1527.434499662" watchObservedRunningTime="2026-11-03 15:33:32.511658341 +0000 UTC m=+1527.436476271"
Nov 03 15:33:34.977577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:34.977531    4955 patch_prober.go:29] interesting pod/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:33:34.978539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:34.978301    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:33:36.165508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.165483    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:33:36.291493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291461    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.291719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291513    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.291719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291549    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.291719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291581    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.291719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291614    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.291719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291698    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.292131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291866    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:36.292131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291880    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.292131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291897    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:36.292131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.291929    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.292131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292037    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs\") pod \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\" (UID: \"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde\") "
Nov 03 15:33:36.292435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292183    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:36.292435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292205    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:33:36.292799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292427    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:36.292941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292918    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.293047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292949    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.293047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.292996    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.293047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.293013    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.293047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.293027    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.294060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.294042    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:33:36.294290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.294275    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "kube-api-access-dr6xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:33:36.294553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.294533    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:33:36.294616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.294547    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy" pod "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" (UID: "1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:33:36.340687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.340633    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf" podStartSLOduration=5.786146426 podStartE2EDuration="9.340618453s" totalImagesPullingTime="3.554472027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:33:32.663714669 +0000 UTC m=+1527.588532595" watchObservedRunningTime="2026-11-03 15:33:36.340618453 +0000 UTC m=+1531.265436365"
Nov 03 15:33:36.395156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.395068    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.395156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.395096    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.395156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.395107    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.395156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.395117    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6xh\" (UniqueName: \"kubernetes.io/projected/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde-kube-api-access-dr6xh\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:33:36.435208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.435172    4955 generic.go:356] "Generic (PLEG): container finished" podID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerID="6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237" exitCode=0
Nov 03 15:33:36.435398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.435218    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"
Nov 03 15:33:36.435398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.435250    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" event={"ID":"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde","Type":"ContainerDied","Data":"6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237"}
Nov 03 15:33:36.435398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.435311    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj" event={"ID":"1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde","Type":"ContainerDied","Data":"00666d865c28747ba92bf47e979ff60178ec570f55706c8a95e1bf3d40cf40f9"}
Nov 03 15:33:36.435398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.435335    4955 scope.go:122] "RemoveContainer" containerID="6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237"
Nov 03 15:33:36.446386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.446369    4955 scope.go:122] "RemoveContainer" containerID="78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"
Nov 03 15:33:36.455367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.455352    4955 scope.go:122] "RemoveContainer" containerID="6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237"
Nov 03 15:33:36.455614 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:36.455596    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237\": container with ID starting with 6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237 not found: ID does not exist" containerID="6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237"
Nov 03 15:33:36.455688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.455622    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237"} err="failed to get container status \"6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237\": rpc error: code = NotFound desc = could not find container \"6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237\": container with ID starting with 6ea1b95f90a7dd05fe81f6056c16f75eb0f029c6c6708c99dfc518cc9c121237 not found: ID does not exist"
Nov 03 15:33:36.455688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.455639    4955 scope.go:122] "RemoveContainer" containerID="78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"
Nov 03 15:33:36.455866 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:36.455851    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14\": container with ID starting with 78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14 not found: ID does not exist" containerID="78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"
Nov 03 15:33:36.455916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.455869    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14"} err="failed to get container status \"78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14\": rpc error: code = NotFound desc = could not find container \"78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14\": container with ID starting with 78d7dcf07b7efd337d68a59b802e27e54be5955950837465efaa62014b272b14 not found: ID does not exist"
Nov 03 15:33:36.603983 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.603941    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"]
Nov 03 15:33:36.608715 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:36.608688    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cceaf27f-openshift-default-5c5f66875f-bmxgj"]
Nov 03 15:33:37.207485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:37.207454    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" path="/var/lib/kubelet/pods/1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde/volumes"
Nov 03 15:33:43.391938 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:43.391909    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:33:43.392344 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:43.392013    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:33:45.293512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:33:45.293476    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:33:57.942867 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:33:57.942830    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:34:03.005398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:03.005356    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"]
Nov 03 15:34:03.005995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:03.005631    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" containerID="cri-o://78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4" gracePeriod=30
Nov 03 15:34:06.098217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.098185    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"]
Nov 03 15:34:06.098926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.098910    4955 cpu_manager.go:414] "containerMap: removing container" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy"
Nov 03 15:34:06.099002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.098928    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy"
Nov 03 15:34:06.099002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.098945    4955 cpu_manager.go:414] "containerMap: removing container" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy"
Nov 03 15:34:06.099002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.098951    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy"
Nov 03 15:34:06.099108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.099077    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy"
Nov 03 15:34:06.099108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.099087    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e10d4bc-c3ae-4d9e-aac2-66cc7e78ccde" containerName="istio-proxy"
Nov 03 15:34:06.105582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.105562    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.158851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.158814    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dfec2d6c-serving-cert\""
Nov 03 15:34:06.162604 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.162570    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"]
Nov 03 15:34:06.218706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.218666    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.218913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.218807    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.218913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.218906    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9rd\" (UniqueName: \"kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.219116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.218993    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.321102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.321069    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.321299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.321187    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.321374 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:34:06.321302    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-serving-cert: secret "maas-api-e2e-migrate-dfec2d6c-serving-cert" not found
Nov 03 15:34:06.321374 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:34:06.321373    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls podName:bb5beebd-bb91-4a27-9188-9a1529a76f3d nodeName:}" failed. No retries permitted until 2026-11-03 15:34:06.821351191 +0000 UTC m=+1561.746169101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls") pod "maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" (UID: "bb5beebd-bb91-4a27-9188-9a1529a76f3d") : secret "maas-api-e2e-migrate-dfec2d6c-serving-cert" not found
Nov 03 15:34:06.321550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.321523    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9rd\" (UniqueName: \"kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.321644 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.321596    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.321835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.321814    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.324150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.324129    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.384152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.384060    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9rd\" (UniqueName: \"kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.840411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.840373    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:06.842909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:06.842886    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\") pod \"maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:07.020488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:07.020452    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:07.405606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:07.405573    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"]
Nov 03 15:34:07.406225 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:07.406200    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5beebd_bb91_4a27_9188_9a1529a76f3d.slice/crio-ffa934030ed36e7dd882b66c7ae92e9b49b7f78b4a0b68c845eec62289d8fe10 WatchSource:0}: Error finding container ffa934030ed36e7dd882b66c7ae92e9b49b7f78b4a0b68c845eec62289d8fe10: Status 404 returned error can't find the container with id ffa934030ed36e7dd882b66c7ae92e9b49b7f78b4a0b68c845eec62289d8fe10
Nov 03 15:34:07.565116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:07.565077    4955 patch_prober.go:29] interesting pod/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 15:34:07.565289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:07.565135    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:34:07.864122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:07.864039    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" event={"ID":"bb5beebd-bb91-4a27-9188-9a1529a76f3d","Type":"ContainerStarted","Data":"ffa934030ed36e7dd882b66c7ae92e9b49b7f78b4a0b68c845eec62289d8fe10"}
Nov 03 15:34:08.398701 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.398668    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:34:08.402750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.402729    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.402830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.402760    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.402830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.402782    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.402909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.402857    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.402956 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.402939    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.403053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403029    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.403120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403073    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.403120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403026    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:34:08.403120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403067    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:34:08.403275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403136    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:34:08.403275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403218    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.403275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403266    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:34:08.403437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403305    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9\") pod \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\" (UID: \"50cee55e-2573-4c85-9ca4-b50a3a5e669a\") "
Nov 03 15:34:08.403437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.403394    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:34:08.404519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.404497    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.404519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.404516    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.404519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.404524    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.404741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.404533    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.404741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.404542    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.405318 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.405298    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:34:08.405461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.405434    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:34:08.405929 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.405834    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:34:08.406427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.406404    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9" pod "50cee55e-2573-4c85-9ca4-b50a3a5e669a" (UID: "50cee55e-2573-4c85-9ca4-b50a3a5e669a"). InnerVolumeSpecName "kube-api-access-9xtf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:34:08.506097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.506064    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.506097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.506092    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.506097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.506101    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xtf9\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-kube-api-access-9xtf9\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.506335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.506111    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cee55e-2573-4c85-9ca4-b50a3a5e669a-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:34:08.875447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.875407    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" event={"ID":"bb5beebd-bb91-4a27-9188-9a1529a76f3d","Type":"ContainerStarted","Data":"e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb"}
Nov 03 15:34:08.875655 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.875510    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:08.877565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.877541    4955 generic.go:356] "Generic (PLEG): container finished" podID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerID="78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4" exitCode=0
Nov 03 15:34:08.877708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.877581    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"
Nov 03 15:34:08.877708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.877627    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" event={"ID":"50cee55e-2573-4c85-9ca4-b50a3a5e669a","Type":"ContainerDied","Data":"78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4"}
Nov 03 15:34:08.877708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.877687    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v" event={"ID":"50cee55e-2573-4c85-9ca4-b50a3a5e669a","Type":"ContainerDied","Data":"e5abda46e0fd5962cad52ca8b3eff163b4277ef0a98e4585c223182f18fb3204"}
Nov 03 15:34:08.877870 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.877718    4955 scope.go:122] "RemoveContainer" containerID="78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4"
Nov 03 15:34:08.888250 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.888229    4955 scope.go:122] "RemoveContainer" containerID="5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"
Nov 03 15:34:08.897420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.897402    4955 scope.go:122] "RemoveContainer" containerID="78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4"
Nov 03 15:34:08.897699 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:34:08.897680    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4\": container with ID starting with 78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4 not found: ID does not exist" containerID="78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4"
Nov 03 15:34:08.897746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.897709    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4"} err="failed to get container status \"78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4\": rpc error: code = NotFound desc = could not find container \"78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4\": container with ID starting with 78c69fec75ed42b77e21fcd2ddcea77b67975d4797b8fe7a0d6392de672902a4 not found: ID does not exist"
Nov 03 15:34:08.897746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.897727    4955 scope.go:122] "RemoveContainer" containerID="5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"
Nov 03 15:34:08.897956 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:34:08.897938    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79\": container with ID starting with 5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79 not found: ID does not exist" containerID="5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"
Nov 03 15:34:08.898053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.897999    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79"} err="failed to get container status \"5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79\": rpc error: code = NotFound desc = could not find container \"5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79\": container with ID starting with 5d37b4d91665862fffdbafa8fea2bb57ad622a7565c45d00dd9d1df573f51c79 not found: ID does not exist"
Nov 03 15:34:08.993709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:08.993649    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" podStartSLOduration=2.07498765 podStartE2EDuration="2.993634028s" totalImagesPullingTime="918.646378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:08.993430162 +0000 UTC m=+1563.918248076" watchObservedRunningTime="2026-11-03 15:34:08.993634028 +0000 UTC m=+1563.918451931"
Nov 03 15:34:09.109268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:09.109230    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"]
Nov 03 15:34:09.114999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:09.113829    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-77d0e91e-openshift-default-58484ff9b5-nw84v"]
Nov 03 15:34:09.272993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:09.272422    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" path="/var/lib/kubelet/pods/50cee55e-2573-4c85-9ca4-b50a3a5e669a/volumes"
Nov 03 15:34:14.892117 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:14.892036    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:34:17.252999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.252950    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"]
Nov 03 15:34:17.253592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.253578    4955 cpu_manager.go:414] "containerMap: removing container" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy"
Nov 03 15:34:17.253592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.253593    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy"
Nov 03 15:34:17.253680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.253619    4955 cpu_manager.go:414] "containerMap: removing container" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy"
Nov 03 15:34:17.253680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.253624    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy"
Nov 03 15:34:17.253743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.253723    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy"
Nov 03 15:34:17.263826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.263800    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.327802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.327766    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"]
Nov 03 15:34:17.361494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361463    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361514    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361593    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361667    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361767    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdb8\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361816    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361909    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.361955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.361936    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.362075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.362059    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.372727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.372703    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-42595854-openshift-default-dockercfg-85gc8\""
Nov 03 15:34:17.477246 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.477203    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.479164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.479137    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.480382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.480360    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.481041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.481021    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdb8\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.481616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.481595    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.482069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.482049    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.482523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.482506    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.483185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.483166    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.483735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.483719    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.484309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.480814    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.484448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.482931    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.484542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.480224    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.484630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.483594    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.484719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.482446    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.495851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.495820    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.517440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.517371    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.571001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.561585    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.571251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.569092    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdb8\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8\") pod \"e2e-mt-42595854-openshift-default-8476dc6c85-txfts\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") " pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:17.598996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:17.598888    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:18.204121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:18.204090    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"]
Nov 03 15:34:18.204238 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:18.204112    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99202fd8_8caa_48a7_9a5f_df882c1098eb.slice/crio-0237fb6c941f01b2a74018436bf4eac591513158758ef5e19aff54991c9e242f WatchSource:0}: Error finding container 0237fb6c941f01b2a74018436bf4eac591513158758ef5e19aff54991c9e242f: Status 404 returned error can't find the container with id 0237fb6c941f01b2a74018436bf4eac591513158758ef5e19aff54991c9e242f
Nov 03 15:34:18.206679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:18.206648    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:18.206742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:18.206716    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:18.206777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:18.206743    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:18.996488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:18.996447    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" event={"ID":"99202fd8-8caa-48a7-9a5f-df882c1098eb","Type":"ContainerStarted","Data":"9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"}
Nov 03 15:34:18.996902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:18.996514    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" event={"ID":"99202fd8-8caa-48a7-9a5f-df882c1098eb","Type":"ContainerStarted","Data":"0237fb6c941f01b2a74018436bf4eac591513158758ef5e19aff54991c9e242f"}
Nov 03 15:34:19.170332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:19.170281    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podStartSLOduration=2.170267451 podStartE2EDuration="2.170267451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:19.16805127 +0000 UTC m=+1574.092869184" watchObservedRunningTime="2026-11-03 15:34:19.170267451 +0000 UTC m=+1574.095085428"
Nov 03 15:34:19.599292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:19.599259    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:19.604674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:19.604650    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:20.005115 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:20.005083    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:20.006192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:20.006171    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:34:25.933995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:25.932528    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"]
Nov 03 15:34:25.977456 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:25.974298    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cee55e-2573-4c85-9ca4-b50a3a5e669a" containerName="istio-proxy"
Nov 03 15:34:25.994259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:25.993401    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"]
Nov 03 15:34:25.994259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:25.993525    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.020028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.003063    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"]
Nov 03 15:34:26.057996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.028267    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.095425    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.114246    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g62ck\" (UniqueName: \"kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck\") pod \"payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.114362    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gvd\" (UniqueName: \"kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd\") pod \"payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") " pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.114613    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume\") pod \"payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") " pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.099580    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-42595854\""
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.101615    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-42595854-dockercfg-4rx47\""
Nov 03 15:34:26.114958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.123786    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"]
Nov 03 15:34:26.234160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.232057    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.234160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.232126    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g62ck\" (UniqueName: \"kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck\") pod \"payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.234160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.232232    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gvd\" (UniqueName: \"kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd\") pod \"payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") " pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.234160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.232465    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume\") pod \"payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") " pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.234160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.233657    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.235581 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.234899    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume\") pod \"payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") " pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.355991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.355476    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g62ck\" (UniqueName: \"kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck\") pod \"payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.372312 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.372044    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:26.375691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.375540    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gvd\" (UniqueName: \"kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd\") pod \"payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") " pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:26.469251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:26.468593    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:27.003122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.003091    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"]
Nov 03 15:34:27.003741 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:27.003716    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cef74ee_4103_48b7_9d2d_0530b73d24e1.slice/crio-46709fb5512c2a9e954a943ab0a6ac66420463d0df59ec5a45843365c23d374c WatchSource:0}: Error finding container 46709fb5512c2a9e954a943ab0a6ac66420463d0df59ec5a45843365c23d374c: Status 404 returned error can't find the container with id 46709fb5512c2a9e954a943ab0a6ac66420463d0df59ec5a45843365c23d374c
Nov 03 15:34:27.055956 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.055929    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"]
Nov 03 15:34:27.056460 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:27.056435    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4f4d06_c595_46c2_88f6_1fe037fbff20.slice/crio-74b34bddf2231ebf2955bfb194a1d8bbc21155ea367da0d396a6e407babd5df1 WatchSource:0}: Error finding container 74b34bddf2231ebf2955bfb194a1d8bbc21155ea367da0d396a6e407babd5df1: Status 404 returned error can't find the container with id 74b34bddf2231ebf2955bfb194a1d8bbc21155ea367da0d396a6e407babd5df1
Nov 03 15:34:27.120560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.120527    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp" event={"ID":"5cef74ee-4103-48b7-9d2d-0530b73d24e1","Type":"ContainerStarted","Data":"4384e74ed0719693204a0e2eb73bc299e8c7a91f25bfa431fa6b00b6f7de040a"}
Nov 03 15:34:27.120707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.120590    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp" event={"ID":"5cef74ee-4103-48b7-9d2d-0530b73d24e1","Type":"ContainerStarted","Data":"46709fb5512c2a9e954a943ab0a6ac66420463d0df59ec5a45843365c23d374c"}
Nov 03 15:34:27.120707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.120618    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:27.122289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.122259    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx" event={"ID":"2b4f4d06-c595-46c2-88f6-1fe037fbff20","Type":"ContainerStarted","Data":"a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f"}
Nov 03 15:34:27.122366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.122303    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx" event={"ID":"2b4f4d06-c595-46c2-88f6-1fe037fbff20","Type":"ContainerStarted","Data":"74b34bddf2231ebf2955bfb194a1d8bbc21155ea367da0d396a6e407babd5df1"}
Nov 03 15:34:27.122411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.122371    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:27.241916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.241855    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp" podStartSLOduration=2.241839475 podStartE2EDuration="2.241839475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:27.240883849 +0000 UTC m=+1582.165701754" watchObservedRunningTime="2026-11-03 15:34:27.241839475 +0000 UTC m=+1582.166657385"
Nov 03 15:34:27.400714 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.400649    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx" podStartSLOduration=2.4006319 podStartE2EDuration="2.4006319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:27.380813668 +0000 UTC m=+1582.305631584" watchObservedRunningTime="2026-11-03 15:34:27.4006319 +0000 UTC m=+1582.325449813"
Nov 03 15:34:27.944858 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:34:27.944821    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:34:27.952147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:27.952117    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:34:31.139062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.129018    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"]
Nov 03 15:34:31.146547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.143374    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.199997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.199069    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-8a0e5a-openshift-default-dockercfg-j6tm7\""
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.376924    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377059    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7klj\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377123    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377174    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377202    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377305    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377358    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377530    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.379999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.377649    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.508002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.503934    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.871687    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.871843    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7klj\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.871923    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.872354    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.872395    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.872527    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.877803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.872602    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.902994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.894614    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"]
Nov 03 15:34:31.902994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.894692    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.902994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.506674    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.923200 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.922930    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.939399 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.939356    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.941165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.939902    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.941338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.940761    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.967995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.967066    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.967995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.967297    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:31.970610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:31.970581    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7klj\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:32.001621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:32.001584    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token\") pod \"e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") " pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:32.088663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:32.088626    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:32.558282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:32.558181    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"]
Nov 03 15:34:32.589006 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:32.588954    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b13cbb_e8e7_4f76_a198_53105c6b4ab0.slice/crio-0ded5ca581b95ff9e4b11c5b01fbd1c18222a95a50a9f1a521d7bac5d4dc7996 WatchSource:0}: Error finding container 0ded5ca581b95ff9e4b11c5b01fbd1c18222a95a50a9f1a521d7bac5d4dc7996: Status 404 returned error can't find the container with id 0ded5ca581b95ff9e4b11c5b01fbd1c18222a95a50a9f1a521d7bac5d4dc7996
Nov 03 15:34:32.591677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:32.591639    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:32.591742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:32.591717    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:32.591786 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:32.591748    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:33.258853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:33.258820    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" event={"ID":"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0","Type":"ContainerStarted","Data":"1ec61e9595c9a67566273d6a68a33e402ae0fc669a4f738193ec2bd04a11ef71"}
Nov 03 15:34:33.259116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:33.258889    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" event={"ID":"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0","Type":"ContainerStarted","Data":"0ded5ca581b95ff9e4b11c5b01fbd1c18222a95a50a9f1a521d7bac5d4dc7996"}
Nov 03 15:34:33.453994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:33.449132    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podStartSLOduration=2.4491081550000002 podStartE2EDuration="2.449108155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:33.431903966 +0000 UTC m=+1588.356721891" watchObservedRunningTime="2026-11-03 15:34:33.449108155 +0000 UTC m=+1588.373926071"
Nov 03 15:34:34.089718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:34.089684    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:34.095380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:34.095356    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:34.266626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:34.266593    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:34.267603 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:34.267581    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:34:38.111109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:38.111071    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:34:38.111602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:38.111141    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:34:38.132769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:38.132741    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:34:38.132931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:38.132810    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:34:39.305733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:39.305694    4955 generic.go:356] "Generic (PLEG): container finished" podID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerID="084ab27cfed9de62e59da6bb95329235bdddc4fb606069855d5d264dffd6de77" exitCode=0
Nov 03 15:34:39.306196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:39.305860    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" event={"ID":"c673af27-2d27-46a3-a386-0a29dbb07d72","Type":"ContainerDied","Data":"084ab27cfed9de62e59da6bb95329235bdddc4fb606069855d5d264dffd6de77"}
Nov 03 15:34:39.306551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:39.306517    4955 scope.go:122] "RemoveContainer" containerID="084ab27cfed9de62e59da6bb95329235bdddc4fb606069855d5d264dffd6de77"
Nov 03 15:34:39.308638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:39.308599    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:39.308824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:39.308767    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:39.308900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:39.308827    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:34:40.211993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.204044    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"]
Nov 03 15:34:40.229006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.225310    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.290005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.289280    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-8a0e5a-dockercfg-ctwv9\""
Nov 03 15:34:40.297004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.296337    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-8a0e5a\""
Nov 03 15:34:40.346849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.346820    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"]
Nov 03 15:34:40.352549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.352516    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"]
Nov 03 15:34:40.380692 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.380200    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.380692 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.380380    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllm6\" (UniqueName: \"kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6\") pod \"payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.383546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.383514    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.567135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.548709    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.567135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.548766    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllm6\" (UniqueName: \"kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6\") pod \"payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.570178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.568226    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tb9m\" (UniqueName: \"kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m\") pod \"payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.570178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.568390    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.602308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.602278    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.719302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.714589    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" event={"ID":"c673af27-2d27-46a3-a386-0a29dbb07d72","Type":"ContainerStarted","Data":"e2217567928da9ede638a60d454d90d1cc1532b68378f66004bfa0f9baf4fe5a"}
Nov 03 15:34:40.719302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.717415    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.719302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.717662    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb9m\" (UniqueName: \"kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m\") pod \"payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.719302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.719195    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.731900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.731862    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"]
Nov 03 15:34:40.748998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.748946    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllm6\" (UniqueName: \"kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6\") pod \"payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:40.800186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.799954    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb9m\" (UniqueName: \"kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m\") pod \"payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.808527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.808494    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:40.892293 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.892264    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:34:40.892913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.892890    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 15:34:40.893002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:40.892938    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 15:34:41.004997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.004941    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:41.227899 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:41.227868    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3e5f81_f567_4156_b6ac_981a1d962408.slice/crio-bdefc8dfa12b95222d92195ca413ec474d871331ee7aac1151411ca8692029b2 WatchSource:0}: Error finding container bdefc8dfa12b95222d92195ca413ec474d871331ee7aac1151411ca8692029b2: Status 404 returned error can't find the container with id bdefc8dfa12b95222d92195ca413ec474d871331ee7aac1151411ca8692029b2
Nov 03 15:34:41.228874 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.228853    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"]
Nov 03 15:34:41.419933 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.419901    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"]
Nov 03 15:34:41.420328 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:34:41.420124    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1c4cfe_7955_43ec_939d_450c1333a38c.slice/crio-65fa8838efa1cccbaeb2b7bd318665b0131e49520a35e3d3246bf27e6873fce2 WatchSource:0}: Error finding container 65fa8838efa1cccbaeb2b7bd318665b0131e49520a35e3d3246bf27e6873fce2: Status 404 returned error can't find the container with id 65fa8838efa1cccbaeb2b7bd318665b0131e49520a35e3d3246bf27e6873fce2
Nov 03 15:34:41.722782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.722687    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds" event={"ID":"fb3e5f81-f567-4156-b6ac-981a1d962408","Type":"ContainerStarted","Data":"44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23"}
Nov 03 15:34:41.722782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.722748    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:41.722782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.722771    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds" event={"ID":"fb3e5f81-f567-4156-b6ac-981a1d962408","Type":"ContainerStarted","Data":"bdefc8dfa12b95222d92195ca413ec474d871331ee7aac1151411ca8692029b2"}
Nov 03 15:34:41.726484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.724902    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r" event={"ID":"5c1c4cfe-7955-43ec-939d-450c1333a38c","Type":"ContainerStarted","Data":"20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43"}
Nov 03 15:34:41.726484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.725025    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r" event={"ID":"5c1c4cfe-7955-43ec-939d-450c1333a38c","Type":"ContainerStarted","Data":"65fa8838efa1cccbaeb2b7bd318665b0131e49520a35e3d3246bf27e6873fce2"}
Nov 03 15:34:41.726484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.726188    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:41.858878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.858820    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds" podStartSLOduration=1.85880694 podStartE2EDuration="1.85880694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:41.853409202 +0000 UTC m=+1596.778227116" watchObservedRunningTime="2026-11-03 15:34:41.85880694 +0000 UTC m=+1596.783624853"
Nov 03 15:34:41.892509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.892470    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 15:34:41.892677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.892536    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 15:34:41.993048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:41.992776    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r" podStartSLOduration=1.992737539 podStartE2EDuration="1.992737539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:34:41.975279766 +0000 UTC m=+1596.900097680" watchObservedRunningTime="2026-11-03 15:34:41.992737539 +0000 UTC m=+1596.917555457"
Nov 03 15:34:42.892890 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:42.892861    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 15:34:42.893315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:42.892924    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 15:34:43.892782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:43.892750    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 15:34:43.892997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:43.892812    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 15:34:44.892367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:44.892322    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 15:34:44.892546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:44.892389    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 15:34:45.892706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:45.892667    4955 patch_prober.go:29] interesting pod/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 15:34:45.893228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:45.892744    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 15:34:46.897199 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:46.897168    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:34:46.897652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:46.897632    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:34:46.898421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:46.898401    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:34:52.759046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:52.759007    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:34:53.767923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:34:53.767892    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:34:57.941733 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:34:57.941696    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:35:06.424520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.424486    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"]
Nov 03 15:35:06.430459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.430435    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.488715 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.488686    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"]
Nov 03 15:35:06.531179 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.531142    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.531367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.531333    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzh9\" (UniqueName: \"kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.531474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.531461    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.531633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.531612    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.533632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.533603    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-42595854-serving-cert\""
Nov 03 15:35:06.632678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.632633    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.632851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.632770    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzh9\" (UniqueName: \"kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.632851 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:06.632778    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-42595854-serving-cert: secret "maas-api-e2e-mt-42595854-serving-cert" not found
Nov 03 15:35:06.632851 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:06.632834    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls podName:1e71cdc8-f7ca-4f55-999d-4487a2648e3b nodeName:}" failed. No retries permitted until 2026-11-03 15:35:07.132817705 +0000 UTC m=+1622.057635623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls") pod "maas-api-e2e-mt-42595854-6d8dd475db-fct9z" (UID: "1e71cdc8-f7ca-4f55-999d-4487a2648e3b") : secret "maas-api-e2e-mt-42595854-serving-cert" not found
Nov 03 15:35:06.633114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.632911    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.633114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.633059    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.633707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.633684    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.635610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.635588    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:06.710596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:06.710481    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzh9\" (UniqueName: \"kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:07.147225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:07.147190    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:07.149800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:07.149779    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\") pod \"maas-api-e2e-mt-42595854-6d8dd475db-fct9z\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:07.346346 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:07.346308    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:07.727154 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:07.727120    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"]
Nov 03 15:35:07.727523 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:35:07.727465    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e71cdc8_f7ca_4f55_999d_4487a2648e3b.slice/crio-2d4c7fed65baba875e628e243856cba8aeecda491dc5b4930b1b670b7fbd5bbb WatchSource:0}: Error finding container 2d4c7fed65baba875e628e243856cba8aeecda491dc5b4930b1b670b7fbd5bbb: Status 404 returned error can't find the container with id 2d4c7fed65baba875e628e243856cba8aeecda491dc5b4930b1b670b7fbd5bbb
Nov 03 15:35:07.729090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:07.729074    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:35:07.978707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:07.978624    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z" event={"ID":"1e71cdc8-f7ca-4f55-999d-4487a2648e3b","Type":"ContainerStarted","Data":"2d4c7fed65baba875e628e243856cba8aeecda491dc5b4930b1b670b7fbd5bbb"}
Nov 03 15:35:08.987377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:08.987343    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z" event={"ID":"1e71cdc8-f7ca-4f55-999d-4487a2648e3b","Type":"ContainerStarted","Data":"0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1"}
Nov 03 15:35:08.987775 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:08.987534    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:09.113770 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:09.113718    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z" podStartSLOduration=2.26295419 podStartE2EDuration="3.113704673s" totalImagesPullingTime="850.750483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:35:09.111705116 +0000 UTC m=+1624.036523030" watchObservedRunningTime="2026-11-03 15:35:09.113704673 +0000 UTC m=+1624.038522585"
Nov 03 15:35:13.414124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:13.414086    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"]
Nov 03 15:35:13.522121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:13.522088    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"]
Nov 03 15:35:15.039996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:15.034522    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:35:18.414568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.414519    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf" podUID="3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" containerName="payload-processing" containerID="cri-o://4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58" gracePeriod=25
Nov 03 15:35:18.691817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.691791    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:35:18.803675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.803643    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume\") pod \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") "
Nov 03 15:35:18.803675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.803680    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l\" (UniqueName: \"kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l\") pod \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\" (UID: \"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc\") "
Nov 03 15:35:18.804025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.804002    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume" pod "3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" (UID: "3ef82c8a-7949-45fd-a44e-6a5ed579c9fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:18.804171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.804156    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:18.805998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.805959    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l" pod "3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" (UID: "3ef82c8a-7949-45fd-a44e-6a5ed579c9fc"). InnerVolumeSpecName "kube-api-access-6k55l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:18.905724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:18.905692    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k55l\" (UniqueName: \"kubernetes.io/projected/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc-kube-api-access-6k55l\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:19.038132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.038024    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr" podUID="26ec121d-7419-4578-af73-45cb28d8c86f" containerName="payload-pre-processing" containerID="cri-o://1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4" gracePeriod=25
Nov 03 15:35:19.128372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.128340    4955 generic.go:356] "Generic (PLEG): container finished" podID="3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" containerID="4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58" exitCode=0
Nov 03 15:35:19.128559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.128387    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"
Nov 03 15:35:19.128559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.128421    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf" event={"ID":"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc","Type":"ContainerDied","Data":"4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58"}
Nov 03 15:35:19.128559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.128475    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf" event={"ID":"3ef82c8a-7949-45fd-a44e-6a5ed579c9fc","Type":"ContainerDied","Data":"e11f73032b290fc15b9477cda9522fb061a70ecda15d74c611544713a1146746"}
Nov 03 15:35:19.128559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.128500    4955 scope.go:122] "RemoveContainer" containerID="4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58"
Nov 03 15:35:19.150849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.150794    4955 scope.go:122] "RemoveContainer" containerID="4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58"
Nov 03 15:35:19.151214 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:19.151187    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58\": container with ID starting with 4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58 not found: ID does not exist" containerID="4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58"
Nov 03 15:35:19.151270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.151225    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58"} err="failed to get container status \"4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58\": rpc error: code = NotFound desc = could not find container \"4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58\": container with ID starting with 4a418b499f36a61fd20eac50826612533b7e42bd40d85ce03f3f20a016534f58 not found: ID does not exist"
Nov 03 15:35:19.300601 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.300575    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:35:19.312522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.312494    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"]
Nov 03 15:35:19.316292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.316264    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfec2d6c-669fc58b87-mrbnf"]
Nov 03 15:35:19.413711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.413676    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume\") pod \"26ec121d-7419-4578-af73-45cb28d8c86f\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") "
Nov 03 15:35:19.413942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.413924    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg\" (UniqueName: \"kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg\") pod \"26ec121d-7419-4578-af73-45cb28d8c86f\" (UID: \"26ec121d-7419-4578-af73-45cb28d8c86f\") "
Nov 03 15:35:19.414158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.414128    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume" pod "26ec121d-7419-4578-af73-45cb28d8c86f" (UID: "26ec121d-7419-4578-af73-45cb28d8c86f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:19.414615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.414594    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ec121d-7419-4578-af73-45cb28d8c86f-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:19.416221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.416202    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg" pod "26ec121d-7419-4578-af73-45cb28d8c86f" (UID: "26ec121d-7419-4578-af73-45cb28d8c86f"). InnerVolumeSpecName "kube-api-access-wqzdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:19.515921 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:19.515887    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzdg\" (UniqueName: \"kubernetes.io/projected/26ec121d-7419-4578-af73-45cb28d8c86f-kube-api-access-wqzdg\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:20.136539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.136501    4955 generic.go:356] "Generic (PLEG): container finished" podID="26ec121d-7419-4578-af73-45cb28d8c86f" containerID="1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4" exitCode=0
Nov 03 15:35:20.136740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.136553    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"
Nov 03 15:35:20.136740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.136585    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr" event={"ID":"26ec121d-7419-4578-af73-45cb28d8c86f","Type":"ContainerDied","Data":"1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4"}
Nov 03 15:35:20.136740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.136647    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr" event={"ID":"26ec121d-7419-4578-af73-45cb28d8c86f","Type":"ContainerDied","Data":"bd19c23feb3c04cab7d5930858fd4d97356daf726e1375344e5297745ef7c269"}
Nov 03 15:35:20.136740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.136663    4955 scope.go:122] "RemoveContainer" containerID="1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4"
Nov 03 15:35:20.147515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.147489    4955 scope.go:122] "RemoveContainer" containerID="1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4"
Nov 03 15:35:20.147804 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:20.147787    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4\": container with ID starting with 1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4 not found: ID does not exist" containerID="1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4"
Nov 03 15:35:20.147848 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.147813    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4"} err="failed to get container status \"1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4\": rpc error: code = NotFound desc = could not find container \"1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4\": container with ID starting with 1c398b4f436d8da9b1400fccf4791c079f6c1e5b7f0645a1bd5d4502414899c4 not found: ID does not exist"
Nov 03 15:35:20.305343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.305314    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"]
Nov 03 15:35:20.355086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:20.355054    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-dfec2d6c-7d7dcfc7dc-rflhr"]
Nov 03 15:35:21.207856 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:21.207823    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ec121d-7419-4578-af73-45cb28d8c86f" path="/var/lib/kubelet/pods/26ec121d-7419-4578-af73-45cb28d8c86f/volumes"
Nov 03 15:35:21.208347 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:21.208330    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" path="/var/lib/kubelet/pods/3ef82c8a-7949-45fd-a44e-6a5ed579c9fc/volumes"
Nov 03 15:35:26.022833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.022796    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"]
Nov 03 15:35:26.023567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.023552    4955 cpu_manager.go:414] "containerMap: removing container" podUID="26ec121d-7419-4578-af73-45cb28d8c86f" containerName="payload-pre-processing"
Nov 03 15:35:26.023626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.023569    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ec121d-7419-4578-af73-45cb28d8c86f" containerName="payload-pre-processing"
Nov 03 15:35:26.023626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.023589    4955 cpu_manager.go:414] "containerMap: removing container" podUID="3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" containerName="payload-processing"
Nov 03 15:35:26.023626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.023595    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" containerName="payload-processing"
Nov 03 15:35:26.023741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.023732    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ec121d-7419-4578-af73-45cb28d8c86f" containerName="payload-pre-processing"
Nov 03 15:35:26.023776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.023742    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef82c8a-7949-45fd-a44e-6a5ed579c9fc" containerName="payload-processing"
Nov 03 15:35:26.027453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.027434    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.081915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.081880    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-8a0e5a-serving-cert\""
Nov 03 15:35:26.125760 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.125726    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.125914 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.125781    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.126022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.125933    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4s52\" (UniqueName: \"kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.126163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.126140    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.136103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.136077    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"]
Nov 03 15:35:26.227875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.227838    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.228146 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.228047    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.228146 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.228100    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.228322 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:26.228053    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-serving-cert: secret "maas-api-e2e-shared-a-w5-8a0e5a-serving-cert" not found
Nov 03 15:35:26.228322 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:26.228234    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls podName:336bce2a-2274-4867-b792-ea1e99b8e4a1 nodeName:}" failed. No retries permitted until 2026-11-03 15:35:26.728211491 +0000 UTC m=+1641.653029399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls") pod "maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" (UID: "336bce2a-2274-4867-b792-ea1e99b8e4a1") : secret "maas-api-e2e-shared-a-w5-8a0e5a-serving-cert" not found
Nov 03 15:35:26.228438 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.228326    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4s52\" (UniqueName: \"kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.228831 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.228810    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.230935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.230913    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.304365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.304179    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4s52\" (UniqueName: \"kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.756183 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.756151    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.758739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.758713    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:26.941172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:26.941142    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:27.323658 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:35:27.323628    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336bce2a_2274_4867_b792_ea1e99b8e4a1.slice/crio-9ee6ffade021ef2bb6ec36568e4f7e21227e1d055141232bcf460f97841c792b WatchSource:0}: Error finding container 9ee6ffade021ef2bb6ec36568e4f7e21227e1d055141232bcf460f97841c792b: Status 404 returned error can't find the container with id 9ee6ffade021ef2bb6ec36568e4f7e21227e1d055141232bcf460f97841c792b
Nov 03 15:35:27.324406 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:27.324367    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"]
Nov 03 15:35:27.945032 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:27.944999    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:35:28.203582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:28.203542    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" event={"ID":"336bce2a-2274-4867-b792-ea1e99b8e4a1","Type":"ContainerStarted","Data":"9ee6ffade021ef2bb6ec36568e4f7e21227e1d055141232bcf460f97841c792b"}
Nov 03 15:35:29.285450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:29.285407    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" event={"ID":"336bce2a-2274-4867-b792-ea1e99b8e4a1","Type":"ContainerStarted","Data":"cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf"}
Nov 03 15:35:29.294934 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:29.287278    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:29.409663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:29.409601    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" podStartSLOduration=3.6090966289999997 podStartE2EDuration="4.409584789s" totalImagesPullingTime="800.48816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:35:29.406922414 +0000 UTC m=+1644.331740327" watchObservedRunningTime="2026-11-03 15:35:29.409584789 +0000 UTC m=+1644.334402701"
Nov 03 15:35:36.304246 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:36.304213    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:35:39.782811 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.782751    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"]
Nov 03 15:35:39.815492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.814329    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:39.886313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.886276    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5cad072f\""
Nov 03 15:35:39.892993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.886695    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5cad072f-dockercfg-6sv7w\""
Nov 03 15:35:39.996116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.923856    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ssn\" (UniqueName: \"kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:39.996116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.930936    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"]
Nov 03 15:35:39.996116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:39.937544    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:40.074095 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:40.053770    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:40.108528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:40.108493    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:40.110266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:40.108848    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ssn\" (UniqueName: \"kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:40.431022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:40.429297    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ssn\" (UniqueName: \"kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn\") pod \"payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:40.457546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:40.457510    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:40.853615 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:35:40.853585    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd280704_5597_4533_8446_3b9a126f9483.slice/crio-c3754475131b4710d65c56d1a56b9cfaf8e4e500ad8e412ce9683f55e27837dc WatchSource:0}: Error finding container c3754475131b4710d65c56d1a56b9cfaf8e4e500ad8e412ce9683f55e27837dc: Status 404 returned error can't find the container with id c3754475131b4710d65c56d1a56b9cfaf8e4e500ad8e412ce9683f55e27837dc
Nov 03 15:35:40.854000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:40.853915    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"]
Nov 03 15:35:41.473749 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:41.473707    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" event={"ID":"bd280704-5597-4533-8446-3b9a126f9483","Type":"ContainerStarted","Data":"d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b"}
Nov 03 15:35:41.473932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:41.473769    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" event={"ID":"bd280704-5597-4533-8446-3b9a126f9483","Type":"ContainerStarted","Data":"c3754475131b4710d65c56d1a56b9cfaf8e4e500ad8e412ce9683f55e27837dc"}
Nov 03 15:35:41.473932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:41.473809    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:41.592821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:41.592761    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" podStartSLOduration=2.592746734 podStartE2EDuration="2.592746734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:35:41.58884495 +0000 UTC m=+1656.513662864" watchObservedRunningTime="2026-11-03 15:35:41.592746734 +0000 UTC m=+1656.517564646"
Nov 03 15:35:45.708843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:45.708766    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"]
Nov 03 15:35:45.709266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:45.708995    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" podUID="bb5beebd-bb91-4a27-9188-9a1529a76f3d" containerName="maas-api" containerID="cri-o://e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb" gracePeriod=30
Nov 03 15:35:46.396995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.391315    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:35:46.514649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.514563    4955 generic.go:356] "Generic (PLEG): container finished" podID="bb5beebd-bb91-4a27-9188-9a1529a76f3d" containerID="e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb" exitCode=0
Nov 03 15:35:46.514649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.514630    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"
Nov 03 15:35:46.514838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.514648    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" event={"ID":"bb5beebd-bb91-4a27-9188-9a1529a76f3d","Type":"ContainerDied","Data":"e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb"}
Nov 03 15:35:46.514838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.514709    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n" event={"ID":"bb5beebd-bb91-4a27-9188-9a1529a76f3d","Type":"ContainerDied","Data":"ffa934030ed36e7dd882b66c7ae92e9b49b7f78b4a0b68c845eec62289d8fe10"}
Nov 03 15:35:46.514838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.514733    4955 scope.go:122] "RemoveContainer" containerID="e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb"
Nov 03 15:35:46.527525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.527502    4955 scope.go:122] "RemoveContainer" containerID="e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb"
Nov 03 15:35:46.527916 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:46.527884    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb\": container with ID starting with e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb not found: ID does not exist" containerID="e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb"
Nov 03 15:35:46.528098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.527926    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb"} err="failed to get container status \"e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb\": rpc error: code = NotFound desc = could not find container \"e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb\": container with ID starting with e6eecbd05b4c30d0c98212e01ed43d21693b6cc07c7f0cd316469fda574159bb not found: ID does not exist"
Nov 03 15:35:46.555447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.555412    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\") pod \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") "
Nov 03 15:35:46.555656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.555474    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs\") pod \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") "
Nov 03 15:35:46.555738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.555654    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd\" (UniqueName: \"kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd\") pod \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") "
Nov 03 15:35:46.555795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.555747    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca\") pod \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\" (UID: \"bb5beebd-bb91-4a27-9188-9a1529a76f3d\") "
Nov 03 15:35:46.556804 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.556777    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca" pod "bb5beebd-bb91-4a27-9188-9a1529a76f3d" (UID: "bb5beebd-bb91-4a27-9188-9a1529a76f3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:46.559142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.559107    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls" pod "bb5beebd-bb91-4a27-9188-9a1529a76f3d" (UID: "bb5beebd-bb91-4a27-9188-9a1529a76f3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:35:46.559290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.559263    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd" pod "bb5beebd-bb91-4a27-9188-9a1529a76f3d" (UID: "bb5beebd-bb91-4a27-9188-9a1529a76f3d"). InnerVolumeSpecName "kube-api-access-hk9rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:46.559460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.559439    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs" pod "bb5beebd-bb91-4a27-9188-9a1529a76f3d" (UID: "bb5beebd-bb91-4a27-9188-9a1529a76f3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:35:46.658420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.658381    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:46.658420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.658414    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5beebd-bb91-4a27-9188-9a1529a76f3d-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:46.658631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.658429    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk9rd\" (UniqueName: \"kubernetes.io/projected/bb5beebd-bb91-4a27-9188-9a1529a76f3d-kube-api-access-hk9rd\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:46.658631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.658442    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5beebd-bb91-4a27-9188-9a1529a76f3d-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:46.941900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.941861    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"]
Nov 03 15:35:46.991317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:46.991277    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfec2d6c-57f448ddcc-d465n"]
Nov 03 15:35:47.207852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:47.207815    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5beebd-bb91-4a27-9188-9a1529a76f3d" path="/var/lib/kubelet/pods/bb5beebd-bb91-4a27-9188-9a1529a76f3d/volumes"
Nov 03 15:35:48.048722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:48.048691    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"]
Nov 03 15:35:48.049208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:48.048908    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy" containerID="cri-o://e2217567928da9ede638a60d454d90d1cc1532b68378f66004bfa0f9baf4fe5a" gracePeriod=30
Nov 03 15:35:49.947979 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:49.947943    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"]
Nov 03 15:35:49.948641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:49.948626    4955 cpu_manager.go:414] "containerMap: removing container" podUID="bb5beebd-bb91-4a27-9188-9a1529a76f3d" containerName="maas-api"
Nov 03 15:35:49.948684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:49.948641    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5beebd-bb91-4a27-9188-9a1529a76f3d" containerName="maas-api"
Nov 03 15:35:49.948812 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:49.948798    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5beebd-bb91-4a27-9188-9a1529a76f3d" containerName="maas-api"
Nov 03 15:35:49.953909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:49.953891    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.011804 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.006930    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-38ded0-openshift-default-dockercfg-7vjmj\""
Nov 03 15:35:50.062660 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.062624    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"]
Nov 03 15:35:50.071751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.071719    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.071951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.071772    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.071951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.071881    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.072136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.072095    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsbr\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.072220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.072164    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.072416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.072392    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.073334 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.073309    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.073455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.073419    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.073514 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.073487    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.238111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.220462    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257027    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.241526    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257313    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257388    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257447    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257501    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257564    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257609    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.257706    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsbr\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.258333    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.258593    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.258874    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.259115    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.263498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.263011    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.269017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.268918    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.389994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.388053    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsbr\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.423055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.416799    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token\") pod \"e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") " pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.568294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.568205    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:50.767710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.767679    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"]
Nov 03 15:35:50.767954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.767929    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" containerID="cri-o://e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2" gracePeriod=30
Nov 03 15:35:50.772160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.772124    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"]
Nov 03 15:35:50.772367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.772345    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy" containerID="cri-o://152564191ceea7eb358f65f5f080c3e366f66065e40529c21d02410b7e00c102" gracePeriod=30
Nov 03 15:35:50.817638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.817603    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"]
Nov 03 15:35:50.817830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.817810    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy" containerID="cri-o://cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949" gracePeriod=30
Nov 03 15:35:50.956330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.956298    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"]
Nov 03 15:35:50.956681 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:35:50.956501    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168c6154_e58d_4ca2_86e3_4893f1b7d9ae.slice/crio-a4d9c957ac3163b84fdbf3107ac5dafd77c7bb10ac601bc310395fafaa8c8c08 WatchSource:0}: Error finding container a4d9c957ac3163b84fdbf3107ac5dafd77c7bb10ac601bc310395fafaa8c8c08: Status 404 returned error can't find the container with id a4d9c957ac3163b84fdbf3107ac5dafd77c7bb10ac601bc310395fafaa8c8c08
Nov 03 15:35:50.959170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.959139    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:35:50.959226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.959202    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:35:50.959257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:50.959230    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:35:51.588230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:51.588191    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" event={"ID":"168c6154-e58d-4ca2-86e3-4893f1b7d9ae","Type":"ContainerStarted","Data":"19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"}
Nov 03 15:35:51.588440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:51.588252    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" event={"ID":"168c6154-e58d-4ca2-86e3-4893f1b7d9ae","Type":"ContainerStarted","Data":"a4d9c957ac3163b84fdbf3107ac5dafd77c7bb10ac601bc310395fafaa8c8c08"}
Nov 03 15:35:51.711122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:51.711069    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podStartSLOduration=2.711051587 podStartE2EDuration="2.711051587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:35:51.709992138 +0000 UTC m=+1666.634810045" watchObservedRunningTime="2026-11-03 15:35:51.711051587 +0000 UTC m=+1666.635869500"
Nov 03 15:35:52.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:52.487005    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:35:52.568783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:52.568749    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:52.574680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:52.574652    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:52.595805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:52.595774    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:52.596947 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:52.596924    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:35:53.373954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:53.373851    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:35:53.871136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:53.871098    4955 generic.go:356] "Generic (PLEG): container finished" podID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerID="e2217567928da9ede638a60d454d90d1cc1532b68378f66004bfa0f9baf4fe5a" exitCode=0
Nov 03 15:35:53.890092 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:53.890047    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" event={"ID":"c673af27-2d27-46a3-a386-0a29dbb07d72","Type":"ContainerDied","Data":"e2217567928da9ede638a60d454d90d1cc1532b68378f66004bfa0f9baf4fe5a"}
Nov 03 15:35:53.890361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:53.890343    4955 scope.go:122] "RemoveContainer" containerID="084ab27cfed9de62e59da6bb95329235bdddc4fb606069855d5d264dffd6de77"
Nov 03 15:35:54.036242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.036135    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:35:54.188502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188463    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.188708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188531    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.188708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188618    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.188835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188754    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.188835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188786    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.188835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188792    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:54.189026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188840    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.189026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188865    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:54.189026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.188883    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.189152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.189088    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:54.189152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.189133    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.189269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.189180    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:54.189269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.189187    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket\") pod \"c673af27-2d27-46a3-a386-0a29dbb07d72\" (UID: \"c673af27-2d27-46a3-a386-0a29dbb07d72\") "
Nov 03 15:35:54.189372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.189355    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:54.190240 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.190204    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.190240 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.190241    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.190428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.190256    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.190428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.190272    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c673af27-2d27-46a3-a386-0a29dbb07d72-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.190428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.190286    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.191216 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.191183    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:35:54.191332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.191293    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:54.191332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.191312    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:54.191448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.191357    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c" pod "c673af27-2d27-46a3-a386-0a29dbb07d72" (UID: "c673af27-2d27-46a3-a386-0a29dbb07d72"). InnerVolumeSpecName "kube-api-access-v2v6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:54.292260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.292222    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.292260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.292258    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2v6c\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-kube-api-access-v2v6c\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.292260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.292269    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.292525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.292278    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c673af27-2d27-46a3-a386-0a29dbb07d72-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:54.891493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.891457    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"
Nov 03 15:35:54.891868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.891786    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb" event={"ID":"c673af27-2d27-46a3-a386-0a29dbb07d72","Type":"ContainerDied","Data":"9fbe9ec171768b2724eb9f09b6db7b32eebe7aa030b5a036ec4bdf4bed30c478"}
Nov 03 15:35:54.891868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:54.891833    4955 scope.go:122] "RemoveContainer" containerID="e2217567928da9ede638a60d454d90d1cc1532b68378f66004bfa0f9baf4fe5a"
Nov 03 15:35:55.115057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.115005    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"]
Nov 03 15:35:55.122586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.122554    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-dfec2d6c-gw-openshift-default-55f676dc89-pbhkb"]
Nov 03 15:35:55.208509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.208432    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" path="/var/lib/kubelet/pods/c673af27-2d27-46a3-a386-0a29dbb07d72/volumes"
Nov 03 15:35:55.582044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.581420    4955 patch_prober.go:29] interesting pod/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:35:55.582044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.581496    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:35:55.907223 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.907188    4955 generic.go:356] "Generic (PLEG): container finished" podID="5248002c-1350-4d29-8e36-68a0a2c75328" containerID="152564191ceea7eb358f65f5f080c3e366f66065e40529c21d02410b7e00c102" exitCode=0
Nov 03 15:35:55.907689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.907241    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" event={"ID":"5248002c-1350-4d29-8e36-68a0a2c75328","Type":"ContainerDied","Data":"152564191ceea7eb358f65f5f080c3e366f66065e40529c21d02410b7e00c102"}
Nov 03 15:35:55.907689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:55.907303    4955 scope.go:122] "RemoveContainer" containerID="49f4998248b3370a34c1dde9dc739b1b1e458f7de8e7f6dc3aea14a5e86a0a21"
Nov 03 15:35:56.307489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.307465    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:35:56.359996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.359495    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.359996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.359592    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.359996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.359659    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.359996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.359775    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.359996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.359810    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.359996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.359953    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.360478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.360079    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.360478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.360180    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.360478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.360270    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl\") pod \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\" (UID: \"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d\") "
Nov 03 15:35:56.393057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.391048    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.393057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.391219    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:56.393057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.391502    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.393057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.391686    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.393057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.391853    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.398006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.397937    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.398386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.398364    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "kube-api-access-2pkkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:56.398469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.398437    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:56.400768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.400716    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo" pod "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" (UID: "29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489532    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489568    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489582    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489597    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pkkl\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-kube-api-access-2pkkl\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489611    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489625    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489639    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489652    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.490038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.489666    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.501702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.501675    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:35:56.510106 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.510015    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595486    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595594    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595651    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595700    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595735    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595795    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.596081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.595828    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.596573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.596232    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.596573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.596331    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.596689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.596652    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.596894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.596777    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.596894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.596821    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.597284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597121    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.597284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597131    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.597740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597396    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.597740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597450    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.597740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597509    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.597740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597565    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.597740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597628    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.597740 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.597657    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.598324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.598116    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:56.599114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.598466    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.599114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.598547    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data\") pod \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\" (UID: \"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55\") "
Nov 03 15:35:56.599114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.598579    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo\") pod \"5248002c-1350-4d29-8e36-68a0a2c75328\" (UID: \"5248002c-1350-4d29-8e36-68a0a2c75328\") "
Nov 03 15:35:56.599114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.598845    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:35:56.599444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599203    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "kube-api-access-7hs28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:56.599500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599464    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:56.599500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599472    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.599627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599609    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599633    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599648    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599663    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5248002c-1350-4d29-8e36-68a0a2c75328-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599662    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.599683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599680    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599695    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599709    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hs28\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-kube-api-access-7hs28\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599727    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.599922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599833    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:35:56.599922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.599873    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.600208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.600008    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:56.600321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.600233    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy" pod "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" (UID: "64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.600580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.600548    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "kube-api-access-rq9mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:35:56.601439 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.601421    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:35:56.601954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.601933    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo" pod "5248002c-1350-4d29-8e36-68a0a2c75328" (UID: "5248002c-1350-4d29-8e36-68a0a2c75328"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710229    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710263    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9mc\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-kube-api-access-rq9mc\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710279    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710293    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710307    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710321    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710335    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5248002c-1350-4d29-8e36-68a0a2c75328-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710348    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710362    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5248002c-1350-4d29-8e36-68a0a2c75328-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:56.710401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:56.710376    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5248002c-1350-4d29-8e36-68a0a2c75328-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:35:57.005186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.003542    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"
Nov 03 15:35:57.005186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.003600    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct" event={"ID":"5248002c-1350-4d29-8e36-68a0a2c75328","Type":"ContainerDied","Data":"d9c9154f11663d81ddd82733a37a65cc27c316b8eb9335725d4196dbc0706a8f"}
Nov 03 15:35:57.005186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.003683    4955 scope.go:122] "RemoveContainer" containerID="152564191ceea7eb358f65f5f080c3e366f66065e40529c21d02410b7e00c102"
Nov 03 15:35:57.015821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.007955    4955 generic.go:356] "Generic (PLEG): container finished" podID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerID="e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2" exitCode=0
Nov 03 15:35:57.015821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.008254    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" event={"ID":"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d","Type":"ContainerDied","Data":"e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2"}
Nov 03 15:35:57.015821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.008311    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c" event={"ID":"29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d","Type":"ContainerDied","Data":"0771830762b561f42dabd91776f67a193344f3275e704d0f0b82d2df24c61ee5"}
Nov 03 15:35:57.015821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.008387    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"
Nov 03 15:35:57.018594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.017748    4955 generic.go:356] "Generic (PLEG): container finished" podID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerID="cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949" exitCode=0
Nov 03 15:35:57.018594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.017825    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" event={"ID":"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55","Type":"ContainerDied","Data":"cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949"}
Nov 03 15:35:57.018594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.017874    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t" event={"ID":"64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55","Type":"ContainerDied","Data":"4f2c457e1ed607ac856dd1e5f91d5caf990cddd9cb469cc13e07442c7ab6ed3c"}
Nov 03 15:35:57.018594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.017960    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"
Nov 03 15:35:57.049259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.045669    4955 scope.go:122] "RemoveContainer" containerID="e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2"
Nov 03 15:35:57.101945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.101920    4955 scope.go:122] "RemoveContainer" containerID="e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2"
Nov 03 15:35:57.102350 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:57.102323    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2\": container with ID starting with e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2 not found: ID does not exist" containerID="e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2"
Nov 03 15:35:57.102449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.102365    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2"} err="failed to get container status \"e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2\": rpc error: code = NotFound desc = could not find container \"e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2\": container with ID starting with e46320d3e35fa58e66af9ce0d9af7c1c110cc077dc32a9f9c007baa3225f5de2 not found: ID does not exist"
Nov 03 15:35:57.102449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.102391    4955 scope.go:122] "RemoveContainer" containerID="cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949"
Nov 03 15:35:57.113787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.113767    4955 scope.go:122] "RemoveContainer" containerID="cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949"
Nov 03 15:35:57.114245 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:57.114199    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949\": container with ID starting with cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949 not found: ID does not exist" containerID="cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949"
Nov 03 15:35:57.114326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.114252    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949"} err="failed to get container status \"cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949\": rpc error: code = NotFound desc = could not find container \"cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949\": container with ID starting with cd2f0c826cf2b7a40dba4b0b2d6a27d6aed7bd0dd9fed22df9a27e8d526ef949 not found: ID does not exist"
Nov 03 15:35:57.222246 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.222199    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"]
Nov 03 15:35:57.226898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.226872    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-rdb7c"]
Nov 03 15:35:57.377032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.365024    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"]
Nov 03 15:35:57.391318 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.391284    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-s8t8t"]
Nov 03 15:35:57.485742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.485710    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"]
Nov 03 15:35:57.492412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:57.492327    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f59d26-openshift-default-7c69db4bb9-tsqct"]
Nov 03 15:35:58.002752 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:35:58.002715    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.407680    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"]
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410191    4955 cpu_manager.go:414] "containerMap: removing container" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410213    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410271    4955 cpu_manager.go:414] "containerMap: removing container" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410279    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410306    4955 cpu_manager.go:414] "containerMap: removing container" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410315    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410334    4955 cpu_manager.go:414] "containerMap: removing container" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410341    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410358    4955 cpu_manager.go:414] "containerMap: removing container" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410364    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410655    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410737    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410752    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410765    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410777    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy"
Nov 03 15:35:58.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.410788    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" containerName="istio-proxy"
Nov 03 15:35:59.000993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.988722    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"]
Nov 03 15:35:59.000993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:58.988855    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.065041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.046295    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-38ded0-dockercfg-gk64h\""
Nov 03 15:35:59.120943 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.102142    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-38ded0\""
Nov 03 15:35:59.120943 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.103158    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"]
Nov 03 15:35:59.120943 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.104338    4955 cpu_manager.go:414] "containerMap: removing container" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy"
Nov 03 15:35:59.120943 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.104358    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c673af27-2d27-46a3-a386-0a29dbb07d72" containerName="istio-proxy"
Nov 03 15:35:59.120943 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.104516    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:35:59.154995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.138192    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.154995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.138400    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2t64\" (UniqueName: \"kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64\") pod \"payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.156701 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.155334    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:35:59.214872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.214814    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.250073 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.229535    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d" path="/var/lib/kubelet/pods/29e662a6-fdd8-4e0b-ab1d-194cd4fc3a3d/volumes"
Nov 03 15:35:59.253781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.241043    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2t64\" (UniqueName: \"kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64\") pod \"payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.253781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.252361    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnsr\" (UniqueName: \"kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr\") pod \"payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.253781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.252521    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.253781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.252699    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.253781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.253674    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.293315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.293287    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5248002c-1350-4d29-8e36-68a0a2c75328" path="/var/lib/kubelet/pods/5248002c-1350-4d29-8e36-68a0a2c75328/volumes"
Nov 03 15:35:59.294122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.294104    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55" path="/var/lib/kubelet/pods/64ce1ad7-d9f4-425d-b94b-fc78ebd7cf55/volumes"
Nov 03 15:35:59.294590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.294576    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"]
Nov 03 15:35:59.308242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.308123    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2t64\" (UniqueName: \"kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64\") pod \"payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.378577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.374682    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.381813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.381782    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.382292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.382266    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnsr\" (UniqueName: \"kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr\") pod \"payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.451996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.451441    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:35:59.483652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.483620    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnsr\" (UniqueName: \"kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr\") pod \"payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.613170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.612909    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:35:59.970145 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:35:59.969894    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"]
Nov 03 15:36:00.120708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.120677    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"]
Nov 03 15:36:00.120813 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:36:00.120723    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3f60b7_c7e7_4570_86fe_7c815204c860.slice/crio-99801ab5bb781fd4d897166cd3be940b7824de785da91763bfd2dd3a8696b3f6 WatchSource:0}: Error finding container 99801ab5bb781fd4d897166cd3be940b7824de785da91763bfd2dd3a8696b3f6: Status 404 returned error can't find the container with id 99801ab5bb781fd4d897166cd3be940b7824de785da91763bfd2dd3a8696b3f6
Nov 03 15:36:00.316604 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.316564    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr" event={"ID":"ab3f60b7-c7e7-4570-86fe-7c815204c860","Type":"ContainerStarted","Data":"621bd84c9f07c2db1a8ae42f53abb0784ddb633fd272c12d2f93b1eb148f80dd"}
Nov 03 15:36:00.316764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.316632    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr" event={"ID":"ab3f60b7-c7e7-4570-86fe-7c815204c860","Type":"ContainerStarted","Data":"99801ab5bb781fd4d897166cd3be940b7824de785da91763bfd2dd3a8696b3f6"}
Nov 03 15:36:00.317091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.317072    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:36:00.319033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.319004    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg" event={"ID":"58df5973-7bdc-4d61-bf48-da056f56bdf9","Type":"ContainerStarted","Data":"4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192"}
Nov 03 15:36:00.319151 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.319045    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg" event={"ID":"58df5973-7bdc-4d61-bf48-da056f56bdf9","Type":"ContainerStarted","Data":"75a6aa52d423da10bc884e65212b3a5a507605f45336d02ef4ea4fddd325ca9d"}
Nov 03 15:36:00.320741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.320720    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:36:00.458170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.458114    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr" podStartSLOduration=2.45809798 podStartE2EDuration="2.45809798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:00.450429183 +0000 UTC m=+1675.375247098" watchObservedRunningTime="2026-11-03 15:36:00.45809798 +0000 UTC m=+1675.382915886"
Nov 03 15:36:00.567453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:00.567392    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg" podStartSLOduration=2.5673708140000002 podStartE2EDuration="2.567370814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:00.566663125 +0000 UTC m=+1675.491481040" watchObservedRunningTime="2026-11-03 15:36:00.567370814 +0000 UTC m=+1675.492188728"
Nov 03 15:36:04.546036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:04.545990    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"]
Nov 03 15:36:04.546595 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:04.546258    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy" containerID="cri-o://31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26" gracePeriod=30
Nov 03 15:36:04.601800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:04.601759    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"]
Nov 03 15:36:04.602084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:04.602056    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy" containerID="cri-o://6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7" gracePeriod=30
Nov 03 15:36:04.605594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:04.605123    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"]
Nov 03 15:36:04.605594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:04.605348    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" containerID="cri-o://19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed" gracePeriod=30
Nov 03 15:36:05.309180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.309144    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:36:05.310195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.309231    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:36:05.371508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.371367    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": read tcp 10.132.0.2:60146->10.132.0.96:15021: read: connection reset by peer" start-of-body=
Nov 03 15:36:05.371508 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.371427    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": read tcp 10.132.0.2:60146->10.132.0.96:15021: read: connection reset by peer"
Nov 03 15:36:05.433753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.433655    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"]
Nov 03 15:36:05.434282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.434246    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy" containerID="cri-o://5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae" gracePeriod=30
Nov 03 15:36:05.441790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.441765    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"]
Nov 03 15:36:05.442027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.442001    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" containerID="cri-o://64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074" gracePeriod=30
Nov 03 15:36:05.483121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.483083    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"]
Nov 03 15:36:05.483335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.483314    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" containerID="cri-o://9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c" gracePeriod=30
Nov 03 15:36:05.489428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.489398    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"]
Nov 03 15:36:05.489802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:05.489774    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" containerID="cri-o://f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1" gracePeriod=30
Nov 03 15:36:06.377536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.377505    4955 generic.go:356] "Generic (PLEG): container finished" podID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerID="1ec61e9595c9a67566273d6a68a33e402ae0fc669a4f738193ec2bd04a11ef71" exitCode=0
Nov 03 15:36:06.378048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.377577    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" event={"ID":"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0","Type":"ContainerDied","Data":"1ec61e9595c9a67566273d6a68a33e402ae0fc669a4f738193ec2bd04a11ef71"}
Nov 03 15:36:06.378134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.378118    4955 scope.go:122] "RemoveContainer" containerID="1ec61e9595c9a67566273d6a68a33e402ae0fc669a4f738193ec2bd04a11ef71"
Nov 03 15:36:06.379436 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.379413    4955 generic.go:356] "Generic (PLEG): container finished" podID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerID="9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab" exitCode=0
Nov 03 15:36:06.379544 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.379486    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" event={"ID":"99202fd8-8caa-48a7-9a5f-df882c1098eb","Type":"ContainerDied","Data":"9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"}
Nov 03 15:36:06.379937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.379920    4955 scope.go:122] "RemoveContainer" containerID="9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"
Nov 03 15:36:06.380006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.379954    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:06.380075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.380058    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:06.380122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.380107    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:06.381444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.381418    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:06.381522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.381483    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:06.381522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.381513    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:06.725478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.725435    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:36:06.725696 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.725503    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:36:06.956203 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.956150    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:36:06.956425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:06.956218    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:36:07.390764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:07.390716    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" event={"ID":"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0","Type":"ContainerStarted","Data":"1b73cd089ad8ea5a792d99cad2603b38f7c44a05dd9eeb7717072934fffb5712"}
Nov 03 15:36:07.393229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:07.393201    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" event={"ID":"99202fd8-8caa-48a7-9a5f-df882c1098eb","Type":"ContainerStarted","Data":"bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2"}
Nov 03 15:36:07.599801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:07.599769    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:36:07.600000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:07.599941    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 15:36:07.600062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:07.600009    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 15:36:08.089056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:08.089021    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:36:08.089374 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:08.089349    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 15:36:08.089429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:08.089392    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 15:36:08.599926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:08.599893    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 15:36:08.600376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:08.599951    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 15:36:09.096628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:09.096593    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 15:36:09.096957 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:09.096910    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 15:36:09.600115 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:09.600085    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 15:36:09.601178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:09.600147    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 15:36:09.890280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:09.888128    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 15:36:09.890280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:09.888190    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 15:36:10.089156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.089090    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 15:36:10.089156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.089145    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 15:36:10.095301 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.093378    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.183589    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.183778    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.183882    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.183936    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.183987    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.184029    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.184111    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.184201    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.190038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.184260    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy\") pod \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\" (UID: \"04dce39a-4460-4fbb-a3c3-d640b4718a0f\") "
Nov 03 15:36:10.192781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.192116    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:10.192781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.192508    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.193055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.193033    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.193263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.193244    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.198201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.198169    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.210027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.205096    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.210027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.205526    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:10.219596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.219565    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:10.219726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.219645    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2" pod "04dce39a-4460-4fbb-a3c3-d640b4718a0f" (UID: "04dce39a-4460-4fbb-a3c3-d640b4718a0f"). InnerVolumeSpecName "kube-api-access-ml4m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:10.228019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.227963    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:36:10.287035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287012    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287214    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287310 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287301    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287374    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287449    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4m2\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-kube-api-access-ml4m2\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287507    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287575    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287642    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dce39a-4460-4fbb-a3c3-d640b4718a0f-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.287717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.287709    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dce39a-4460-4fbb-a3c3-d640b4718a0f-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.292820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.292792    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:36:10.383049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.376163    4955 patch_prober.go:29] interesting pod/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:36:10.383049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.376228    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391409    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391482    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391513    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391561    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391633    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391719    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391882    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.393164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.391943    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.397459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.397435    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.397618 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.397602    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.397762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.397745    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.397902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.397890    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.406152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406112    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.406780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406525    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.406780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406604    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.406780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406768    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.407074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406818    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.407074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406834    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket\") pod \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\" (UID: \"320e09e5-9f50-4f2c-8d41-e2736054eb7f\") "
Nov 03 15:36:10.407074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.406992    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.407074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407060    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert\") pod \"508f7618-681b-4acf-8651-0805f1de1202\" (UID: \"508f7618-681b-4acf-8651-0805f1de1202\") "
Nov 03 15:36:10.407319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407268    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.407319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407294    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.408174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407469    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.408174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407660    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.408174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407777    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:10.408174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.407872    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.408612 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.408587    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.413705    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.413787    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "kube-api-access-h48tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414095    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414154    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414222    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414243    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414257    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414271    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414294    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414297    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414308    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414322    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414339    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.415093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.414352    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.421705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.417186    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:10.426407 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.425205    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:10.499041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.495701    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl" pod "508f7618-681b-4acf-8651-0805f1de1202" (UID: "508f7618-681b-4acf-8651-0805f1de1202"). InnerVolumeSpecName "kube-api-access-xlxpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:10.504436 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.501956    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo" pod "320e09e5-9f50-4f2c-8d41-e2736054eb7f" (UID: "320e09e5-9f50-4f2c-8d41-e2736054eb7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:10.517254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517211    4955 generic.go:356] "Generic (PLEG): container finished" podID="508f7618-681b-4acf-8651-0805f1de1202" containerID="19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed" exitCode=0
Nov 03 15:36:10.517412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517371    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" event={"ID":"508f7618-681b-4acf-8651-0805f1de1202","Type":"ContainerDied","Data":"19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed"}
Nov 03 15:36:10.517530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517489    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6" event={"ID":"508f7618-681b-4acf-8651-0805f1de1202","Type":"ContainerDied","Data":"61f3e519433aab739853727d89ac0a91faf41899e55d715b2273d73c12e16d62"}
Nov 03 15:36:10.517602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517552    4955 scope.go:122] "RemoveContainer" containerID="19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed"
Nov 03 15:36:10.517602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517569    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517598    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48tf\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-kube-api-access-h48tf\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517616    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508f7618-681b-4acf-8651-0805f1de1202-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517632    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517652    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508f7618-681b-4acf-8651-0805f1de1202-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517666    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517680    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508f7618-681b-4acf-8651-0805f1de1202-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517694    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320e09e5-9f50-4f2c-8d41-e2736054eb7f-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.517718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.517714    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlxpl\" (UniqueName: \"kubernetes.io/projected/508f7618-681b-4acf-8651-0805f1de1202-kube-api-access-xlxpl\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:10.518531 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.518394    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"
Nov 03 15:36:10.524145 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.524116    4955 generic.go:356] "Generic (PLEG): container finished" podID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerID="31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26" exitCode=0
Nov 03 15:36:10.524371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.524347    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" event={"ID":"04dce39a-4460-4fbb-a3c3-d640b4718a0f","Type":"ContainerDied","Data":"31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26"}
Nov 03 15:36:10.524470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.524401    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst" event={"ID":"04dce39a-4460-4fbb-a3c3-d640b4718a0f","Type":"ContainerDied","Data":"f4526678e2323261088d4aac6ecced05544eef30533c64daa1d7161ed714ecb7"}
Nov 03 15:36:10.555994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.537398    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"
Nov 03 15:36:10.606143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.606101    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 15:36:10.606637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.606192    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 15:36:10.614171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.613861    4955 generic.go:356] "Generic (PLEG): container finished" podID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerID="6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7" exitCode=0
Nov 03 15:36:10.614264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.614237    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" event={"ID":"320e09e5-9f50-4f2c-8d41-e2736054eb7f","Type":"ContainerDied","Data":"6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7"}
Nov 03 15:36:10.614326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.614301    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p" event={"ID":"320e09e5-9f50-4f2c-8d41-e2736054eb7f","Type":"ContainerDied","Data":"151752da1ddc020e945184c4633ed95c3fcfd4c32cdf30ad198b9005851d719e"}
Nov 03 15:36:10.614623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.614406    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"
Nov 03 15:36:10.731687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.728856    4955 scope.go:122] "RemoveContainer" containerID="19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed"
Nov 03 15:36:10.732672 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:10.732218    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed\": container with ID starting with 19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed not found: ID does not exist" containerID="19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed"
Nov 03 15:36:10.732672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.732260    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed"} err="failed to get container status \"19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed\": rpc error: code = NotFound desc = could not find container \"19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed\": container with ID starting with 19e6e48e775298cb0e016314fed63d2f374b3751128412e2be38d6a0a9e76bed not found: ID does not exist"
Nov 03 15:36:10.732672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.732287    4955 scope.go:122] "RemoveContainer" containerID="31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26"
Nov 03 15:36:10.887179 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.887140    4955 scope.go:122] "RemoveContainer" containerID="31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26"
Nov 03 15:36:10.888021 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:10.887632    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26\": container with ID starting with 31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26 not found: ID does not exist" containerID="31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26"
Nov 03 15:36:10.888021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.887671    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26"} err="failed to get container status \"31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26\": rpc error: code = NotFound desc = could not find container \"31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26\": container with ID starting with 31f184f55855daefbdd3078c4bbbcc8c7e771e08ca95afe7797dddd7240f5a26 not found: ID does not exist"
Nov 03 15:36:10.888021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.887698    4955 scope.go:122] "RemoveContainer" containerID="6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7"
Nov 03 15:36:10.904412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:10.904316    4955 scope.go:122] "RemoveContainer" containerID="01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"
Nov 03 15:36:11.094463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.094428    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"]
Nov 03 15:36:11.096816 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.096170    4955 scope.go:122] "RemoveContainer" containerID="6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7"
Nov 03 15:36:11.097502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.097286    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 15:36:11.097502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.097344    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 15:36:11.097502 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.097389    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7\": container with ID starting with 6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7 not found: ID does not exist" containerID="6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7"
Nov 03 15:36:11.097502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.097420    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7"} err="failed to get container status \"6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7\": rpc error: code = NotFound desc = could not find container \"6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7\": container with ID starting with 6be104b85254a8f36d14cba0e3b9e6b0a82f93c09fd3590c7f5e92cc22beb4b7 not found: ID does not exist"
Nov 03 15:36:11.097502 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.097443    4955 scope.go:122] "RemoveContainer" containerID="01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"
Nov 03 15:36:11.097851 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.097815    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e\": container with ID starting with 01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e not found: ID does not exist" containerID="01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"
Nov 03 15:36:11.097908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.097844    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e"} err="failed to get container status \"01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e\": rpc error: code = NotFound desc = could not find container \"01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e\": container with ID starting with 01bd775c23707ccbe0697d11f3d954a41439cfb0731a0b4a783851b2a457266e not found: ID does not exist"
Nov 03 15:36:11.100391 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.100362    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-wng7p"]
Nov 03 15:36:11.204211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.204176    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:36:11.210082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.209716    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" path="/var/lib/kubelet/pods/320e09e5-9f50-4f2c-8d41-e2736054eb7f/volumes"
Nov 03 15:36:11.309987 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.309881    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.310274 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.310253    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.310724 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.310705    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.310878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.310864    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.311076 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.311058    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.311220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.311206    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.311395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.311380    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.311582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.311569    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.311741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.311727    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy\") pod \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\" (UID: \"08c86c97-2e4f-468a-9a41-4f1c73a103a6\") "
Nov 03 15:36:11.312293 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.312267    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"]
Nov 03 15:36:11.312710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.311766    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.312857 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.312096    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.312959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.312875    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:11.313181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.313138    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.313289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.313212    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.322483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.322435    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.322873 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.322834    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.323839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.323809    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-zskr6"]
Nov 03 15:36:11.325891 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.325844    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:11.326454 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.326429    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw" pod "08c86c97-2e4f-468a-9a41-4f1c73a103a6" (UID: "08c86c97-2e4f-468a-9a41-4f1c73a103a6"). InnerVolumeSpecName "kube-api-access-7f5qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.402181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.402113    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"]
Nov 03 15:36:11.403291 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.403228    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-7wtst"]
Nov 03 15:36:11.405832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.405042    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414783    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414812    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414829    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414846    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5qw\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-kube-api-access-7f5qw\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414860    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414873    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414886    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414900    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c86c97-2e4f-468a-9a41-4f1c73a103a6-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.415517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.414913    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c86c97-2e4f-468a-9a41-4f1c73a103a6-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.507660 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.507627    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:36:11.509721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.509696    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:36:11.511820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.511796    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:36:11.515902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.515879    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.515910    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.515986    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516012    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516043    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516061    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516130    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516165    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516212    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516231    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516272    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516289    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516284    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516311    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516355 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516332    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516389    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516436    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data\") pod \"620d28d1-d950-453f-9811-0d8f13dd5726\" (UID: \"620d28d1-d950-453f-9811-0d8f13dd5726\") "
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516469    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516476    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516570    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516590    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data\") pod \"7872f25b-a49f-4d04-8f3f-86de12e1811d\" (UID: \"7872f25b-a49f-4d04-8f3f-86de12e1811d\") "
Nov 03 15:36:11.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516837    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.517282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.516950    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.517282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517242    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.517414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517386    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.517414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517414    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.517592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517432    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.517592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517447    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7872f25b-a49f-4d04-8f3f-86de12e1811d-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.517592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517462    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.517592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517476    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.517820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517612    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.517820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.517804    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:11.518083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.518062    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.518333 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.518252    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.519129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.519070    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.519631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.519590    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:11.519631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.519605    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.519948 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.519918    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "kube-api-access-q2757". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.520628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.520607    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "kube-api-access-vtcjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.521067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.520932    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.521067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.521042    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo" pod "620d28d1-d950-453f-9811-0d8f13dd5726" (UID: "620d28d1-d950-453f-9811-0d8f13dd5726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:11.521317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.521264    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token" pod "7872f25b-a49f-4d04-8f3f-86de12e1811d" (UID: "7872f25b-a49f-4d04-8f3f-86de12e1811d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.600233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.599706    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 15:36:11.600233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.599772    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 15:36:11.618370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618337    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618385    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618527    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618574    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618637    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618704    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618774    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618774    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.618840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618839    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.619278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618869    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.619278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.618919    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert\") pod \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\" (UID: \"1a22d35e-eead-4c08-a1ab-b48b074f2d81\") "
Nov 03 15:36:11.619278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619011    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.619278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619152    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.619549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619530    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:11.619622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619604    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619630    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619645    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d28d1-d950-453f-9811-0d8f13dd5726-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619659    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d28d1-d950-453f-9811-0d8f13dd5726-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619673    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619686    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619701    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtcjm\" (UniqueName: \"kubernetes.io/projected/620d28d1-d950-453f-9811-0d8f13dd5726-kube-api-access-vtcjm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619716    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619729    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619742    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7872f25b-a49f-4d04-8f3f-86de12e1811d-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619754    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619766    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619780    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619792    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2757\" (UniqueName: \"kubernetes.io/projected/7872f25b-a49f-4d04-8f3f-86de12e1811d-kube-api-access-q2757\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619805    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.619877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.619822    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d28d1-d950-453f-9811-0d8f13dd5726-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.622019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.621963    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:11.622204 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.622110    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.622602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.622579    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:11.623407 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.623385    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs" pod "1a22d35e-eead-4c08-a1ab-b48b074f2d81" (UID: "1a22d35e-eead-4c08-a1ab-b48b074f2d81"). InnerVolumeSpecName "kube-api-access-mnvhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:11.626281 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.626251    4955 generic.go:356] "Generic (PLEG): container finished" podID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerID="5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae" exitCode=0
Nov 03 15:36:11.626409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.626301    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"
Nov 03 15:36:11.626409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.626323    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" event={"ID":"08c86c97-2e4f-468a-9a41-4f1c73a103a6","Type":"ContainerDied","Data":"5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae"}
Nov 03 15:36:11.626409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.626375    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz" event={"ID":"08c86c97-2e4f-468a-9a41-4f1c73a103a6","Type":"ContainerDied","Data":"8d5a22e211fa1af40820840bfdfee6c5c9f420035e620712c16ded6948391328"}
Nov 03 15:36:11.626565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.626410    4955 scope.go:122] "RemoveContainer" containerID="5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae"
Nov 03 15:36:11.628699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.628596    4955 generic.go:356] "Generic (PLEG): container finished" podID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerID="9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c" exitCode=0
Nov 03 15:36:11.628699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.628671    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"
Nov 03 15:36:11.628877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.628675    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" event={"ID":"7872f25b-a49f-4d04-8f3f-86de12e1811d","Type":"ContainerDied","Data":"9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c"}
Nov 03 15:36:11.628877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.628799    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp" event={"ID":"7872f25b-a49f-4d04-8f3f-86de12e1811d","Type":"ContainerDied","Data":"b940b75de1366a9857faa45de52faab7479c2825903ddc90f50c02606252b8c7"}
Nov 03 15:36:11.721302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.721270    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnvhs\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-kube-api-access-mnvhs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.721464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.721304    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.721464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.721322    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.721464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.721339    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.721464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.721353    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22d35e-eead-4c08-a1ab-b48b074f2d81-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:11.727046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.725448    4955 generic.go:356] "Generic (PLEG): container finished" podID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerID="f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1" exitCode=0
Nov 03 15:36:11.727046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.725568    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"
Nov 03 15:36:11.750035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.738197    4955 scope.go:122] "RemoveContainer" containerID="9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"
Nov 03 15:36:11.783047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.741346    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" event={"ID":"1a22d35e-eead-4c08-a1ab-b48b074f2d81","Type":"ContainerDied","Data":"f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1"}
Nov 03 15:36:11.783047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.755670    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs" event={"ID":"1a22d35e-eead-4c08-a1ab-b48b074f2d81","Type":"ContainerDied","Data":"8135610fa1f20dc63b055e8dff5164d4b8ab468a830708138bd03f2776a98678"}
Nov 03 15:36:11.783047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.755709    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" event={"ID":"620d28d1-d950-453f-9811-0d8f13dd5726","Type":"ContainerDied","Data":"64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074"}
Nov 03 15:36:11.783047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.745059    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"
Nov 03 15:36:11.783047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.744985    4955 generic.go:356] "Generic (PLEG): container finished" podID="620d28d1-d950-453f-9811-0d8f13dd5726" containerID="64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074" exitCode=0
Nov 03 15:36:11.783047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.763098    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2" event={"ID":"620d28d1-d950-453f-9811-0d8f13dd5726","Type":"ContainerDied","Data":"3a9f1866216de1821838847f1a867b5344eb317c0843fac9cfbebd9ee5cbe48f"}
Nov 03 15:36:11.805320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.805188    4955 scope.go:122] "RemoveContainer" containerID="5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae"
Nov 03 15:36:11.805954 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.805700    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae\": container with ID starting with 5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae not found: ID does not exist" containerID="5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae"
Nov 03 15:36:11.805954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.805738    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae"} err="failed to get container status \"5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae\": rpc error: code = NotFound desc = could not find container \"5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae\": container with ID starting with 5c34ac522b0bfa65a22068ab9968380425f62006ad7ebfc5e341b853081c24ae not found: ID does not exist"
Nov 03 15:36:11.805954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.805764    4955 scope.go:122] "RemoveContainer" containerID="9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"
Nov 03 15:36:11.806529 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.806415    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6\": container with ID starting with 9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6 not found: ID does not exist" containerID="9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"
Nov 03 15:36:11.806529 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.806464    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6"} err="failed to get container status \"9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6\": rpc error: code = NotFound desc = could not find container \"9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6\": container with ID starting with 9fef564198df0e169b62a4876fcf562bd1760c266b415118caa395f90b9fdfc6 not found: ID does not exist"
Nov 03 15:36:11.806529 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.806489    4955 scope.go:122] "RemoveContainer" containerID="9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c"
Nov 03 15:36:11.818573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.818474    4955 scope.go:122] "RemoveContainer" containerID="9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c"
Nov 03 15:36:11.818834 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.818815    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c\": container with ID starting with 9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c not found: ID does not exist" containerID="9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c"
Nov 03 15:36:11.818899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.818843    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c"} err="failed to get container status \"9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c\": rpc error: code = NotFound desc = could not find container \"9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c\": container with ID starting with 9b2ed6ece63fe566c28f5d0a8cefe141756c1eab0bf26ca99b9c75e8a31a5f4c not found: ID does not exist"
Nov 03 15:36:11.818899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.818861    4955 scope.go:122] "RemoveContainer" containerID="f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1"
Nov 03 15:36:11.866283 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.862057    4955 scope.go:122] "RemoveContainer" containerID="f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1"
Nov 03 15:36:11.866283 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.862501    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1\": container with ID starting with f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1 not found: ID does not exist" containerID="f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1"
Nov 03 15:36:11.866283 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.862536    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1"} err="failed to get container status \"f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1\": rpc error: code = NotFound desc = could not find container \"f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1\": container with ID starting with f7351fe7076377ddaa5ec04737b0cc87745614eeafe2e08dada599229d1fafa1 not found: ID does not exist"
Nov 03 15:36:11.866283 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.862563    4955 scope.go:122] "RemoveContainer" containerID="64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074"
Nov 03 15:36:11.879270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.878784    4955 scope.go:122] "RemoveContainer" containerID="64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074"
Nov 03 15:36:11.879700 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:11.879586    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074\": container with ID starting with 64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074 not found: ID does not exist" containerID="64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074"
Nov 03 15:36:11.879700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:11.879623    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074"} err="failed to get container status \"64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074\": rpc error: code = NotFound desc = could not find container \"64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074\": container with ID starting with 64c034a3f47da28196700e98df0c88a88f2943afe7dd3eb7b027aa3db0526074 not found: ID does not exist"
Nov 03 15:36:12.091309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.091271    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 15:36:12.091498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.091333    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 15:36:12.301651 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.301611    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"]
Nov 03 15:36:12.304922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.304898    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-2skqs"]
Nov 03 15:36:12.381559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.381529    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:36:12.420996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.420481    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"]
Nov 03 15:36:12.425687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.425658    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-fxmhz"]
Nov 03 15:36:12.565331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.565238    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"]
Nov 03 15:36:12.578486 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.578432    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-qpvn2"]
Nov 03 15:36:12.600233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.600195    4955 patch_prober.go:29] interesting pod/e2e-mt-42595854-openshift-default-8476dc6c85-txfts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 15:36:12.600384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.600258    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 15:36:12.676904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.676860    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"]
Nov 03 15:36:12.678895 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:12.678858    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b841a-openshift-default-77ffd97d4c-dmkjp"]
Nov 03 15:36:13.093624 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.093589    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:36:13.094017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.093998    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:36:13.094785 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.094766    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:36:13.206952 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.206920    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" path="/var/lib/kubelet/pods/04dce39a-4460-4fbb-a3c3-d640b4718a0f/volumes"
Nov 03 15:36:13.207436 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.207421    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" path="/var/lib/kubelet/pods/08c86c97-2e4f-468a-9a41-4f1c73a103a6/volumes"
Nov 03 15:36:13.207903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.207888    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" path="/var/lib/kubelet/pods/1a22d35e-eead-4c08-a1ab-b48b074f2d81/volumes"
Nov 03 15:36:13.208612 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.208597    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508f7618-681b-4acf-8651-0805f1de1202" path="/var/lib/kubelet/pods/508f7618-681b-4acf-8651-0805f1de1202/volumes"
Nov 03 15:36:13.209037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.209022    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" path="/var/lib/kubelet/pods/620d28d1-d950-453f-9811-0d8f13dd5726/volumes"
Nov 03 15:36:13.209430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.209418    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" path="/var/lib/kubelet/pods/7872f25b-a49f-4d04-8f3f-86de12e1811d/volumes"
Nov 03 15:36:13.604318 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.604281    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:36:13.604592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.604575    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:36:13.605315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:13.605292    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:36:27.944160 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:27.944118    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:36:28.890691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.889517    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"]
Nov 03 15:36:28.891492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.891450    4955 cpu_manager.go:414] "containerMap: removing container" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy"
Nov 03 15:36:28.891647 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.891633    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy"
Nov 03 15:36:28.891735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.891725    4955 cpu_manager.go:414] "containerMap: removing container" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy"
Nov 03 15:36:28.891824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.891814    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy"
Nov 03 15:36:28.892010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.891899    4955 cpu_manager.go:414] "containerMap: removing container" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy"
Nov 03 15:36:28.892115 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892102    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy"
Nov 03 15:36:28.892230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892220    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy"
Nov 03 15:36:28.892300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892291    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy"
Nov 03 15:36:28.892570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892548    4955 cpu_manager.go:414] "containerMap: removing container" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy"
Nov 03 15:36:28.892683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892672    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy"
Nov 03 15:36:28.892756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892747    4955 cpu_manager.go:414] "containerMap: removing container" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy"
Nov 03 15:36:28.892821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892812    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy"
Nov 03 15:36:28.892904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892895    4955 cpu_manager.go:414] "containerMap: removing container" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy"
Nov 03 15:36:28.892996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.892985    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy"
Nov 03 15:36:28.893087 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893076    4955 cpu_manager.go:414] "containerMap: removing container" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy"
Nov 03 15:36:28.893178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893165    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy"
Nov 03 15:36:28.893443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893429    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7872f25b-a49f-4d04-8f3f-86de12e1811d" containerName="istio-proxy"
Nov 03 15:36:28.893627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893615    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="620d28d1-d950-453f-9811-0d8f13dd5726" containerName="istio-proxy"
Nov 03 15:36:28.893722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893711    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a22d35e-eead-4c08-a1ab-b48b074f2d81" containerName="istio-proxy"
Nov 03 15:36:28.893801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893792    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy"
Nov 03 15:36:28.894162 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.893914    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy"
Nov 03 15:36:28.894302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.894286    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c86c97-2e4f-468a-9a41-4f1c73a103a6" containerName="istio-proxy"
Nov 03 15:36:28.894398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.894388    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy"
Nov 03 15:36:28.894480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.894468    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="508f7618-681b-4acf-8651-0805f1de1202" containerName="istio-proxy"
Nov 03 15:36:28.894553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.894545    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="04dce39a-4460-4fbb-a3c3-d640b4718a0f" containerName="istio-proxy"
Nov 03 15:36:28.899323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.899303    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:28.941484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.941450    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"]
Nov 03 15:36:28.988124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:28.988094    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5cad072f-serving-cert\""
Nov 03 15:36:29.043079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.043035    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.043267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.043232    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.043449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.043426    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmpvb\" (UniqueName: \"kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.043572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.043560    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.145511 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.145426    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.145675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.145520    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmpvb\" (UniqueName: \"kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.145675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.145622    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.145675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.145664    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.145845 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:29.145786    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-serving-cert: secret "maas-api-e2e-mt-5cad072f-serving-cert" not found
Nov 03 15:36:29.145845 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:29.145838    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls podName:7eddbea6-7716-4cef-89cf-e6f67baaa593 nodeName:}" failed. No retries permitted until 2026-11-03 15:36:29.645822295 +0000 UTC m=+1704.570640185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls") pod "maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" (UID: "7eddbea6-7716-4cef-89cf-e6f67baaa593") : secret "maas-api-e2e-mt-5cad072f-serving-cert" not found
Nov 03 15:36:29.146289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.146257    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.148377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.148360    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.246992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.246882    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmpvb\" (UniqueName: \"kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.659570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.659533    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.662412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.662383    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\") pod \"maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.816306 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.816253    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:29.970057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.964295    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"]
Nov 03 15:36:29.993335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.979190    4955 cpu_manager.go:414] "containerMap: removing container" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy"
Nov 03 15:36:29.993335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.979259    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320e09e5-9f50-4f2c-8d41-e2736054eb7f" containerName="istio-proxy"
Nov 03 15:36:29.993335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:29.988872    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019007    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019118    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019159    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019215    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxwv\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019286    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019347    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019432    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019485    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.021022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.019509    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.047009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.046048    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-7b3fd3-openshift-default-dockercfg-xqrdg\""
Nov 03 15:36:30.097805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.092480    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"]
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127108    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127185    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127258    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127300    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127320    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127382    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127463    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127503    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.127557    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxwv\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.128649    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.128893    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.129447    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.130090    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.142998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.131257    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.214788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.214325    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.218960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.217816    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxwv\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.275046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.274172    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.310877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.299820    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo\") pod \"e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") " pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.314445 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.314403    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"]
Nov 03 15:36:30.457590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.457554    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:30.916482 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.916449    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"]
Nov 03 15:36:30.916826 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:36:30.916800    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691ab01b_5d2e_4dd9_868e_c49f850e2e67.slice/crio-25e1f6aa9ee4bddf39fb93d86b5e589e29d2506b6aad0b523100b5c951615527 WatchSource:0}: Error finding container 25e1f6aa9ee4bddf39fb93d86b5e589e29d2506b6aad0b523100b5c951615527: Status 404 returned error can't find the container with id 25e1f6aa9ee4bddf39fb93d86b5e589e29d2506b6aad0b523100b5c951615527
Nov 03 15:36:30.919492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.919462    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:30.919555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.919526    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:30.919599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.919554    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:36:30.964622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.964588    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" event={"ID":"7eddbea6-7716-4cef-89cf-e6f67baaa593","Type":"ContainerStarted","Data":"0c4fe6ae25afe4dc4e41ac1f7813b7a8ce5b3fd768a60bdbdffcc7ad0d72bb7a"}
Nov 03 15:36:30.966007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:30.965956    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" event={"ID":"691ab01b-5d2e-4dd9-868e-c49f850e2e67","Type":"ContainerStarted","Data":"25e1f6aa9ee4bddf39fb93d86b5e589e29d2506b6aad0b523100b5c951615527"}
Nov 03 15:36:31.974852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:31.974811    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" event={"ID":"7eddbea6-7716-4cef-89cf-e6f67baaa593","Type":"ContainerStarted","Data":"963372608fdd0f995576340afc678da91204212f1a9f36f519415f7e063c4c5a"}
Nov 03 15:36:31.975370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:31.975024    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:31.977357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:31.977331    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" event={"ID":"691ab01b-5d2e-4dd9-868e-c49f850e2e67","Type":"ContainerStarted","Data":"c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"}
Nov 03 15:36:32.096634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:32.096570    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" podStartSLOduration=3.055019507 podStartE2EDuration="4.096554968s" totalImagesPullingTime="1.041535461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:32.094196487 +0000 UTC m=+1707.019014400" watchObservedRunningTime="2026-11-03 15:36:32.096554968 +0000 UTC m=+1707.021372926"
Nov 03 15:36:32.227307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:32.227209    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podStartSLOduration=3.227194995 podStartE2EDuration="3.227194995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:32.223055045 +0000 UTC m=+1707.147872958" watchObservedRunningTime="2026-11-03 15:36:32.227194995 +0000 UTC m=+1707.152012907"
Nov 03 15:36:32.464994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:32.464340    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:32.511453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:32.511368    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:32.984673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:32.984645    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:32.985615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:32.985594    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:36:35.054713 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.054678    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"]
Nov 03 15:36:35.055137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.054867    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy" containerID="cri-o://d1726b39da778f8a4739dc74e1aa9bb78116dc0a839e24124313af271df409b0" gracePeriod=30
Nov 03 15:36:35.105964 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.105927    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"]
Nov 03 15:36:35.106248 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.106199    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" containerID="cri-o://a09c136faf744d2121ad26a03dabce4f2a58afa7482c7e05ec38ac86feb977c4" gracePeriod=30
Nov 03 15:36:35.109268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.109236    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"]
Nov 03 15:36:35.109477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.109452    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" containerID="cri-o://9671b229e36bfe44ea795156c54ab1825f85366807d6846eea92de16bc3a5f04" gracePeriod=30
Nov 03 15:36:35.157097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.157065    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"]
Nov 03 15:36:35.157360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:35.157320    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" containerID="cri-o://5e5496bd0d36e938e9785c7afc840e9af41c6a2ad29ca5256096cb521a4bd00d" gracePeriod=30
Nov 03 15:36:36.691376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:36.691330    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:36:36.691798 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:36.691394    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:36:37.990408 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:37.990381    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:36:38.619052 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.619016    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:36:38.622163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.622125    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:36:38.815446 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.815411    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"]
Nov 03 15:36:38.820853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.820834    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:38.874444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.874325    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-7b3fd3-dockercfg-8hk6b\""
Nov 03 15:36:38.874652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.874625    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-7b3fd3\""
Nov 03 15:36:38.878525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.878498    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"]
Nov 03 15:36:38.929996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.922313    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkk8r\" (UniqueName: \"kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r\") pod \"payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:38.929996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:38.922480    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:39.027997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.027722    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"]
Nov 03 15:36:39.047002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.031081    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkk8r\" (UniqueName: \"kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r\") pod \"payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:39.047002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.031263    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:39.047002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.032632    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:39.047002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.035767    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.152060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.152005    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2cf\" (UniqueName: \"kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf\") pod \"payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.152555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.152525    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.256096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.256055    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.256511 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.256488    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2cf\" (UniqueName: \"kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf\") pod \"payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.258080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.258059    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.262313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.262294    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkk8r\" (UniqueName: \"kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r\") pod \"payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:39.278587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.278551    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"]
Nov 03 15:36:39.322150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.322120    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2cf\" (UniqueName: \"kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf\") pod \"payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.349271 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.349236    4955 patch_prober.go:29] interesting pod/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:36:39.349415 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.349303    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:36:39.360850 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.360814    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:39.435280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.435243    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:39.743997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.743949    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"]
Nov 03 15:36:39.744650 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:36:39.744621    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ad851e_cfa6_4b35_9a9a_b3947af80cad.slice/crio-4eb8a16b8033262a8d2f0af01fa2e7da0fd4b5410f22a3d292ff174e3c089bae WatchSource:0}: Error finding container 4eb8a16b8033262a8d2f0af01fa2e7da0fd4b5410f22a3d292ff174e3c089bae: Status 404 returned error can't find the container with id 4eb8a16b8033262a8d2f0af01fa2e7da0fd4b5410f22a3d292ff174e3c089bae
Nov 03 15:36:39.846896 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:39.846866    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"]
Nov 03 15:36:39.847638 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:36:39.847612    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ee3799_2775_47d9_8589_9ef649bbd2a2.slice/crio-fea6319d16df2b341fd29daba6ae5deef03f84014c9fc34e5fbd020cf6099e5e WatchSource:0}: Error finding container fea6319d16df2b341fd29daba6ae5deef03f84014c9fc34e5fbd020cf6099e5e: Status 404 returned error can't find the container with id fea6319d16df2b341fd29daba6ae5deef03f84014c9fc34e5fbd020cf6099e5e
Nov 03 15:36:40.294918 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.294882    4955 generic.go:356] "Generic (PLEG): container finished" podID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerID="a09c136faf744d2121ad26a03dabce4f2a58afa7482c7e05ec38ac86feb977c4" exitCode=0
Nov 03 15:36:40.296454 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.296067    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" event={"ID":"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5","Type":"ContainerDied","Data":"a09c136faf744d2121ad26a03dabce4f2a58afa7482c7e05ec38ac86feb977c4"}
Nov 03 15:36:40.298135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.298112    4955 generic.go:356] "Generic (PLEG): container finished" podID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerID="d1726b39da778f8a4739dc74e1aa9bb78116dc0a839e24124313af271df409b0" exitCode=0
Nov 03 15:36:40.298288 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.298239    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" event={"ID":"05f98bb0-5fbb-475c-b269-508c01f458f4","Type":"ContainerDied","Data":"d1726b39da778f8a4739dc74e1aa9bb78116dc0a839e24124313af271df409b0"}
Nov 03 15:36:40.299903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.299877    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss" event={"ID":"d8ee3799-2775-47d9-8589-9ef649bbd2a2","Type":"ContainerStarted","Data":"f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024"}
Nov 03 15:36:40.300129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.299927    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss" event={"ID":"d8ee3799-2775-47d9-8589-9ef649bbd2a2","Type":"ContainerStarted","Data":"fea6319d16df2b341fd29daba6ae5deef03f84014c9fc34e5fbd020cf6099e5e"}
Nov 03 15:36:40.300955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.300933    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:40.307301 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.304009    4955 generic.go:356] "Generic (PLEG): container finished" podID="121e1692-9acb-41c7-97b3-44a9db147e46" containerID="9671b229e36bfe44ea795156c54ab1825f85366807d6846eea92de16bc3a5f04" exitCode=0
Nov 03 15:36:40.307301 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.304093    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" event={"ID":"121e1692-9acb-41c7-97b3-44a9db147e46","Type":"ContainerDied","Data":"9671b229e36bfe44ea795156c54ab1825f85366807d6846eea92de16bc3a5f04"}
Nov 03 15:36:40.312990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.310265    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl" event={"ID":"99ad851e-cfa6-4b35-9a9a-b3947af80cad","Type":"ContainerStarted","Data":"b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da"}
Nov 03 15:36:40.312990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.310311    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl" event={"ID":"99ad851e-cfa6-4b35-9a9a-b3947af80cad","Type":"ContainerStarted","Data":"4eb8a16b8033262a8d2f0af01fa2e7da0fd4b5410f22a3d292ff174e3c089bae"}
Nov 03 15:36:40.312990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.311026    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:40.312990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.312737    4955 generic.go:356] "Generic (PLEG): container finished" podID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerID="5e5496bd0d36e938e9785c7afc840e9af41c6a2ad29ca5256096cb521a4bd00d" exitCode=0
Nov 03 15:36:40.312990 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.312779    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" event={"ID":"f809bb82-5978-4c18-9faa-1d0eb992e650","Type":"ContainerDied","Data":"5e5496bd0d36e938e9785c7afc840e9af41c6a2ad29ca5256096cb521a4bd00d"}
Nov 03 15:36:40.424522 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.424437    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss" podStartSLOduration=2.424420087 podStartE2EDuration="2.424420087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:40.419698175 +0000 UTC m=+1715.344516089" watchObservedRunningTime="2026-11-03 15:36:40.424420087 +0000 UTC m=+1715.349237994"
Nov 03 15:36:40.510838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.509813    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:36:40.515332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.515308    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:36:40.547614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.546158    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl" podStartSLOduration=2.546136069 podStartE2EDuration="2.546136069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:40.544579495 +0000 UTC m=+1715.469397409" watchObservedRunningTime="2026-11-03 15:36:40.546136069 +0000 UTC m=+1715.470953985"
Nov 03 15:36:40.578516 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.577863    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:36:40.600834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.600796    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.600860    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.600890    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.600927    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.600987    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601081    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601113    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601146    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601184    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601226    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601266    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601327    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601363    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601393    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601421    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601449    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601494    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601517    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.601654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601625    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.602184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601686    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.602184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601776    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.602184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601879    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.602184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601924    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.602184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.601986    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.607849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.606370    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket\") pod \"121e1692-9acb-41c7-97b3-44a9db147e46\" (UID: \"121e1692-9acb-41c7-97b3-44a9db147e46\") "
Nov 03 15:36:40.607849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.606441    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket\") pod \"05f98bb0-5fbb-475c-b269-508c01f458f4\" (UID: \"05f98bb0-5fbb-475c-b269-508c01f458f4\") "
Nov 03 15:36:40.607849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.606521    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv\") pod \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\" (UID: \"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5\") "
Nov 03 15:36:40.612566 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.612513    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.612776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.612749    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.614139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.614109    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.614349 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.614326    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.614521 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.614501    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.614701 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.614668    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.616828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.616073    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:40.616828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.616432    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.624167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.624123    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.638012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.624482    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.654989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.644229    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.654989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.650679    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.654989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.651361    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:40.656445 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.655997    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.657193 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.656999    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.682360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.674439    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.682360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.682258    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:40.682812 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.682764    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "kube-api-access-lcrjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.683395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.683032    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.715003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.714436    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716365    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716396    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716409    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716422    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716435    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121e1692-9acb-41c7-97b3-44a9db147e46-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716447    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f98bb0-5fbb-475c-b269-508c01f458f4-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716460    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716473    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121e1692-9acb-41c7-97b3-44a9db147e46-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716485    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716498    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716510    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716523    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716535    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716548    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrjv\" (UniqueName: \"kubernetes.io/projected/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-kube-api-access-lcrjv\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716562    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716573    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716588    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716601    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f98bb0-5fbb-475c-b269-508c01f458f4-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716613    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.716999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.716624    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121e1692-9acb-41c7-97b3-44a9db147e46-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.719866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.717743    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "kube-api-access-7vqqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.719866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.717867    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.719866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.717994    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.721380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.721344    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:40.721598 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.721555    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo" pod "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" (UID: "b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:40.722370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.722337    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22" pod "05f98bb0-5fbb-475c-b269-508c01f458f4" (UID: "05f98bb0-5fbb-475c-b269-508c01f458f4"). InnerVolumeSpecName "kube-api-access-ppt22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.722598 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.722568    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token" pod "121e1692-9acb-41c7-97b3-44a9db147e46" (UID: "121e1692-9acb-41c7-97b3-44a9db147e46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.723257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.723179    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.854902    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.854941    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqqg\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-kube-api-access-7vqqg\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.854957    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.854993    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121e1692-9acb-41c7-97b3-44a9db147e46-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.855009    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.855022    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.855492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.855037    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppt22\" (UniqueName: \"kubernetes.io/projected/05f98bb0-5fbb-475c-b269-508c01f458f4-kube-api-access-ppt22\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.955926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.955887    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.956216 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.956032    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.956303 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.956259    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.956495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.956066    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.957027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957008    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.957202 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957181    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.957330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957270    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.957330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957300    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.957449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957398    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.957509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957488    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.957564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957528    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:36:40.957624 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957574    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97\") pod \"f809bb82-5978-4c18-9faa-1d0eb992e650\" (UID: \"f809bb82-5978-4c18-9faa-1d0eb992e650\") "
Nov 03 15:36:40.958142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.957727    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.958242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.958221    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:40.958488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.958460    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.958488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.958488    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f809bb82-5978-4c18-9faa-1d0eb992e650-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.958650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.958504    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.958650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.958518    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.958650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.958531    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:40.960241 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.960219    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "kube-api-access-9kj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.960485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.960464    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:36:40.961719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.961700    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:36:40.961904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:40.961888    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy" pod "f809bb82-5978-4c18-9faa-1d0eb992e650" (UID: "f809bb82-5978-4c18-9faa-1d0eb992e650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:36:41.059477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.059440    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:41.059477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.059471    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:41.059477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.059481    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f809bb82-5978-4c18-9faa-1d0eb992e650-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:41.059717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.059492    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kj97\" (UniqueName: \"kubernetes.io/projected/f809bb82-5978-4c18-9faa-1d0eb992e650-kube-api-access-9kj97\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:36:41.322912 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.321909    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"
Nov 03 15:36:41.322912 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.321918    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5" event={"ID":"f809bb82-5978-4c18-9faa-1d0eb992e650","Type":"ContainerDied","Data":"a6e97d48d7e5fc1fac9bc6c8a48936a9e74a8fb131d2cd0bfe5fa618788b462d"}
Nov 03 15:36:41.322912 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.322036    4955 scope.go:122] "RemoveContainer" containerID="5e5496bd0d36e938e9785c7afc840e9af41c6a2ad29ca5256096cb521a4bd00d"
Nov 03 15:36:41.332401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.332368    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt" event={"ID":"b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5","Type":"ContainerDied","Data":"772327f732070caa2d54e5aa08e2a464a2f0c670bbded37d8b5fe49a583cd3e7"}
Nov 03 15:36:41.332649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.332503    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"
Nov 03 15:36:41.336104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.335600    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt" event={"ID":"05f98bb0-5fbb-475c-b269-508c01f458f4","Type":"ContainerDied","Data":"9788a1bea5b66c5961cc2b82d680b69e0849d906048ac2722f53d7de8acd3d1b"}
Nov 03 15:36:41.336104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.335740    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"
Nov 03 15:36:41.343835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.343809    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"
Nov 03 15:36:41.344000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.343837    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j" event={"ID":"121e1692-9acb-41c7-97b3-44a9db147e46","Type":"ContainerDied","Data":"4515c50423291b93b1dd40e4b5d1f6a01949162308058abd9c2cf7bb6e577256"}
Nov 03 15:36:41.353071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.353050    4955 scope.go:122] "RemoveContainer" containerID="a09c136faf744d2121ad26a03dabce4f2a58afa7482c7e05ec38ac86feb977c4"
Nov 03 15:36:41.365239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.365219    4955 scope.go:122] "RemoveContainer" containerID="d1726b39da778f8a4739dc74e1aa9bb78116dc0a839e24124313af271df409b0"
Nov 03 15:36:41.377372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.377348    4955 scope.go:122] "RemoveContainer" containerID="9671b229e36bfe44ea795156c54ab1825f85366807d6846eea92de16bc3a5f04"
Nov 03 15:36:41.491430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.491399    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"]
Nov 03 15:36:41.497539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.497463    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-hckdt"]
Nov 03 15:36:41.609267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.609234    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"]
Nov 03 15:36:41.613002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.612977    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lg6p5"]
Nov 03 15:36:41.724902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.724865    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"]
Nov 03 15:36:41.731347 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.731321    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-lz8jt"]
Nov 03 15:36:41.845641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.845555    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"]
Nov 03 15:36:41.853810 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:41.853784    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e62ff3-openshift-default-67c44c944f-j9n5j"]
Nov 03 15:36:43.207389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:43.207355    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" path="/var/lib/kubelet/pods/05f98bb0-5fbb-475c-b269-508c01f458f4/volumes"
Nov 03 15:36:43.207895 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:43.207879    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" path="/var/lib/kubelet/pods/121e1692-9acb-41c7-97b3-44a9db147e46/volumes"
Nov 03 15:36:43.208373 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:43.208360    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" path="/var/lib/kubelet/pods/b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5/volumes"
Nov 03 15:36:43.209161 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:43.209147    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" path="/var/lib/kubelet/pods/f809bb82-5978-4c18-9faa-1d0eb992e650/volumes"
Nov 03 15:36:49.818073 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.818037    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"]
Nov 03 15:36:49.818949 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.818927    4955 cpu_manager.go:414] "containerMap: removing container" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.818952    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.818990    4955 cpu_manager.go:414] "containerMap: removing container" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.818998    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819022    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819029    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819086    4955 cpu_manager.go:414] "containerMap: removing container" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy"
Nov 03 15:36:49.819109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819094    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy"
Nov 03 15:36:49.819442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819258    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f98bb0-5fbb-475c-b269-508c01f458f4" containerName="istio-proxy"
Nov 03 15:36:49.819442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819277    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="121e1692-9acb-41c7-97b3-44a9db147e46" containerName="istio-proxy"
Nov 03 15:36:49.819442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819291    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="f809bb82-5978-4c18-9faa-1d0eb992e650" containerName="istio-proxy"
Nov 03 15:36:49.819442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.819305    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bc3475-4053-49bb-a3c9-7d00fc1e9ac5" containerName="istio-proxy"
Nov 03 15:36:49.824498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.824472    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.882881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.882842    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"]
Nov 03 15:36:49.894321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.894284    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.894509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.894447    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.894509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.894479    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ftq\" (UniqueName: \"kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.894710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.894638    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.928201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.928171    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-38ded0-serving-cert\""
Nov 03 15:36:49.996395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.996356    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.997184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.996904    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.997184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.997059    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.997184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.997098    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ftq\" (UniqueName: \"kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:49.999483 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:49.998781    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-serving-cert: secret "maas-api-e2e-shared-a-w6-38ded0-serving-cert" not found
Nov 03 15:36:49.999483 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:49.998862    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls podName:3c8117ac-048b-48fb-9316-90f39450be94 nodeName:}" failed. No retries permitted until 2026-11-03 15:36:50.498841157 +0000 UTC m=+1725.423659063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls") pod "maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" (UID: "3c8117ac-048b-48fb-9316-90f39450be94") : secret "maas-api-e2e-shared-a-w6-38ded0-serving-cert" not found
Nov 03 15:36:50.004165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:49.999744    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:50.004165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:50.003195    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:50.061872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:50.061837    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ftq\" (UniqueName: \"kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:50.518801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:50.518766    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:50.521411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:50.521384    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:50.738843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:50.738811    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:51.138395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:51.138360    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"]
Nov 03 15:36:51.138823 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:36:51.138763    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8117ac_048b_48fb_9316_90f39450be94.slice/crio-2ff4d4837f2418ec0bb8bb471a79be10c5c6c735d64d361edfd76cd6178c5f04 WatchSource:0}: Error finding container 2ff4d4837f2418ec0bb8bb471a79be10c5c6c735d64d361edfd76cd6178c5f04: Status 404 returned error can't find the container with id 2ff4d4837f2418ec0bb8bb471a79be10c5c6c735d64d361edfd76cd6178c5f04
Nov 03 15:36:51.420680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:51.420645    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" event={"ID":"3c8117ac-048b-48fb-9316-90f39450be94","Type":"ContainerStarted","Data":"2ff4d4837f2418ec0bb8bb471a79be10c5c6c735d64d361edfd76cd6178c5f04"}
Nov 03 15:36:52.357896 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:52.357822    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:36:52.357896 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:52.357895    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:36:52.433011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:52.432943    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" event={"ID":"3c8117ac-048b-48fb-9316-90f39450be94","Type":"ContainerStarted","Data":"1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4"}
Nov 03 15:36:52.433209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:52.433067    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:36:52.729013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:52.728915    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" podStartSLOduration=2.921335701 podStartE2EDuration="3.728896682s" totalImagesPullingTime="807.560981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:36:52.728149614 +0000 UTC m=+1727.652967530" watchObservedRunningTime="2026-11-03 15:36:52.728896682 +0000 UTC m=+1727.653714597"
Nov 03 15:36:57.942902 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:36:57.942863    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:36:58.446031 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:36:58.446003    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:37:12.179365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.179332    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"]
Nov 03 15:37:12.185597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.185558    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271206    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271305    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271343    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271381    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271452    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl96\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271531    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271568    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271701    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.275017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.271757    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.305032 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.297175    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-fe0cb9-openshift-default-dockercfg-hv8zz\""
Nov 03 15:37:12.310479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.306735    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"]
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384407    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384498    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl96\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384576    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384615    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384712    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384768    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384873    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384940    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.384995    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.408997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.385935    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.414340    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.414990    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.421995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.415233    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.442189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.425737    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.442189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.426020    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.442189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.426346    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.542113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.535039    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.542113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.537422    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl96\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96\") pod \"e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") " pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:12.565985 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:12.565883    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:13.095461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.095425    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"]
Nov 03 15:37:13.095890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:37:13.095865    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba150f8_5538_4677_90c9_178795e8c449.slice/crio-a7f4b8b44b5e0a2f4c2ad989c571bfb3bc80c37b96b6191e2e2cc4c42660b21e WatchSource:0}: Error finding container a7f4b8b44b5e0a2f4c2ad989c571bfb3bc80c37b96b6191e2e2cc4c42660b21e: Status 404 returned error can't find the container with id a7f4b8b44b5e0a2f4c2ad989c571bfb3bc80c37b96b6191e2e2cc4c42660b21e
Nov 03 15:37:13.099167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.099125    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:13.099259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.099215    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:13.099259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.099252    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:13.147324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.147283    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:37:13.634524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.634491    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" event={"ID":"4ba150f8-5538-4677-90c9-178795e8c449","Type":"ContainerStarted","Data":"d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"}
Nov 03 15:37:13.635024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.634558    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" event={"ID":"4ba150f8-5538-4677-90c9-178795e8c449","Type":"ContainerStarted","Data":"a7f4b8b44b5e0a2f4c2ad989c571bfb3bc80c37b96b6191e2e2cc4c42660b21e"}
Nov 03 15:37:13.770272 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:13.770198    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podStartSLOduration=1.770182638 podStartE2EDuration="1.770182638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:13.767180462 +0000 UTC m=+1748.691998376" watchObservedRunningTime="2026-11-03 15:37:13.770182638 +0000 UTC m=+1748.695000552"
Nov 03 15:37:14.569070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:14.568430    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:14.603476 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:14.603436    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:14.686997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:14.686702    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:14.699995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:14.691214    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:37:15.413500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.413450    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:15.652141 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.652071    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:15.688174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.688055    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:37:15.688733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.688700    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 03 15:37:15.777116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.777067    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:37:15.778237 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.778196    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:15.926594 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:15.926550    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:16.181674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.181629    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:16.202235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.202185    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 15:37:16.415993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.415092    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:16.415993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.415791    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:37:16.819057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.773515    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:16.819057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.776586    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:37:16.838110 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.836791    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:16.866061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.852367    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:37:16.911136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.911083    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:16.917998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:16.911951    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:37:17.114099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.107780    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:37:17.212673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.212623    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:17.215681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.215654    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:37:17.220460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.220432    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:37:17.418654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.418290    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:37:17.418915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.418892    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:37:17.419058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.418940    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:37:17.419058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.418991    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:37:17.419058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.419024    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:37:17.419058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:17.419054    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:37:19.363103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.363069    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"]
Nov 03 15:37:19.369049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.369028    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.424399 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.424363    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5d3cde-openshift-default-dockercfg-pg6m8\""
Nov 03 15:37:19.477065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477035    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477241 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477069    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477247    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477270    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477300    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477354    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477500    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2h74\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477542    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477611    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.477805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.477675    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"]
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584174    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584236    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584285    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584333    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584401    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2h74\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584428    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584466    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584623    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.584660    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.587018 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.585210    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.590993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.590137    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.601017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.599443    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.601017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.599685    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.601017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.600684    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.682994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.673405    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"]
Nov 03 15:37:19.717075 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.717025    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy" containerID="cri-o://4d0e4fec4598de0bdfbeb48aa45ef884f50f179238353f2dd5855b9917bcafe9" gracePeriod=30
Nov 03 15:37:19.717427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.705496    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2h74\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.726997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.708289    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.767496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.766312    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.909664 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.909617    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo\") pod \"e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") " pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:19.983537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:19.983439    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:20.584523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.584497    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"]
Nov 03 15:37:20.587580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.587536    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:20.587726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.587625    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:20.587726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.587671    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:20.591916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.591895    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"]
Nov 03 15:37:20.592051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.592031    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:20.622534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.622505    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"]
Nov 03 15:37:20.661026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.660992    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-fe0cb9\""
Nov 03 15:37:20.695116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.674919    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-fe0cb9-dockercfg-b9qp4\""
Nov 03 15:37:20.698010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.696539    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:20.698010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.696851    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6wvr\" (UniqueName: \"kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr\") pod \"payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:20.746002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.739344    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"]
Nov 03 15:37:20.784007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.777516    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:20.824994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.800111    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6wvr\" (UniqueName: \"kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr\") pod \"payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:20.849789 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.829656    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:20.849789 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.848373    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:20.863618 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.863583    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"]
Nov 03 15:37:20.980937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.971875    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:20.980937 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.972087    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc\") pod \"payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:20.988225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:20.988157    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6wvr\" (UniqueName: \"kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr\") pod \"payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:21.017661 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.017623    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" event={"ID":"a019f010-05ff-47d9-8a42-bc507ab4780c","Type":"ContainerStarted","Data":"12f052df44439d26c050b909271de51d590306cb2f1b9b2fee145379ae780440"}
Nov 03 15:37:21.017859 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.017701    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" event={"ID":"a019f010-05ff-47d9-8a42-bc507ab4780c","Type":"ContainerStarted","Data":"adb951aabc386b0b511390f50db4c49a4dc5912754cf80f66e0dd0fdc2d3ebdd"}
Nov 03 15:37:21.073020 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.072960    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc\") pod \"payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:21.073197 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.073160    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:21.074107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.074088    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:21.135118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.135050    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc\") pod \"payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:21.142180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.142129    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podStartSLOduration=2.14211333 podStartE2EDuration="2.14211333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:21.139828256 +0000 UTC m=+1756.064646182" watchObservedRunningTime="2026-11-03 15:37:21.14211333 +0000 UTC m=+1756.066931306"
Nov 03 15:37:21.284778 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.282112    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:21.312998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.312280    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:21.963030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.962991    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"]
Nov 03 15:37:21.983867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.983834    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:21.990793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:21.990766    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:22.001378 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:37:22.001343    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c8fe04_de0d_40d4_9cf4_2a8294138819.slice/crio-0b2744ce08e99cce8068a3acdf0765ade113d2f23ecd163fc57f61929786e10f WatchSource:0}: Error finding container 0b2744ce08e99cce8068a3acdf0765ade113d2f23ecd163fc57f61929786e10f: Status 404 returned error can't find the container with id 0b2744ce08e99cce8068a3acdf0765ade113d2f23ecd163fc57f61929786e10f
Nov 03 15:37:22.012264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:22.012239    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"]
Nov 03 15:37:22.018869 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:37:22.018843    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cc575e_ced1_4e36_a805_6b7d3982fe7b.slice/crio-2a29a73c5981f985f2978d7ee223a78f2b9cd8f13c05d2959b11daeef166129a WatchSource:0}: Error finding container 2a29a73c5981f985f2978d7ee223a78f2b9cd8f13c05d2959b11daeef166129a: Status 404 returned error can't find the container with id 2a29a73c5981f985f2978d7ee223a78f2b9cd8f13c05d2959b11daeef166129a
Nov 03 15:37:22.027801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:22.027765    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472" event={"ID":"53c8fe04-de0d-40d4-9cf4-2a8294138819","Type":"ContainerStarted","Data":"0b2744ce08e99cce8068a3acdf0765ade113d2f23ecd163fc57f61929786e10f"}
Nov 03 15:37:22.028315 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:22.028293    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:22.029570 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:22.029549    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:37:23.039044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.039005    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472" event={"ID":"53c8fe04-de0d-40d4-9cf4-2a8294138819","Type":"ContainerStarted","Data":"0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35"}
Nov 03 15:37:23.039457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.039161    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:23.041269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.041238    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw" event={"ID":"44cc575e-ced1-4e36-a805-6b7d3982fe7b","Type":"ContainerStarted","Data":"297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a"}
Nov 03 15:37:23.041427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.041288    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw" event={"ID":"44cc575e-ced1-4e36-a805-6b7d3982fe7b","Type":"ContainerStarted","Data":"2a29a73c5981f985f2978d7ee223a78f2b9cd8f13c05d2959b11daeef166129a"}
Nov 03 15:37:23.158714 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.158650    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472" podStartSLOduration=3.158635947 podStartE2EDuration="3.158635947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:23.158438409 +0000 UTC m=+1758.083256323" watchObservedRunningTime="2026-11-03 15:37:23.158635947 +0000 UTC m=+1758.083453861"
Nov 03 15:37:23.314986 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.308295    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw" podStartSLOduration=3.3082746849999998 podStartE2EDuration="3.308274685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:23.306758863 +0000 UTC m=+1758.231576818" watchObservedRunningTime="2026-11-03 15:37:23.308274685 +0000 UTC m=+1758.233092598"
Nov 03 15:37:23.639279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.639146    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:37:23.639454 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:23.639281    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:37:24.048116 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:24.048088    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:25.056253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.056218    4955 generic.go:356] "Generic (PLEG): container finished" podID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerID="4d0e4fec4598de0bdfbeb48aa45ef884f50f179238353f2dd5855b9917bcafe9" exitCode=0
Nov 03 15:37:25.056741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.056291    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" event={"ID":"0ceeffa5-00cc-4a12-8da7-fd463083e22a","Type":"ContainerDied","Data":"4d0e4fec4598de0bdfbeb48aa45ef884f50f179238353f2dd5855b9917bcafe9"}
Nov 03 15:37:25.056741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.056369    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z" event={"ID":"0ceeffa5-00cc-4a12-8da7-fd463083e22a","Type":"ContainerDied","Data":"72675ee5131d7435901813a8fa5588c8a9e8a80b0c13e43bd667cdededf79b27"}
Nov 03 15:37:25.056741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.056389    4955 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72675ee5131d7435901813a8fa5588c8a9e8a80b0c13e43bd667cdededf79b27"
Nov 03 15:37:25.080461 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.080391    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:37:25.173872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.173831    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.173914    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.173945    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174229    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174263    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174347    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174374    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:37:25.174558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174440    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174476    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174523    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data\") pod \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\" (UID: \"0ceeffa5-00cc-4a12-8da7-fd463083e22a\") "
Nov 03 15:37:25.174821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.174692    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:37:25.175939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.175502    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.175939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.175529    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.175939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.175634    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:37:25.175939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.175823    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:37:25.175939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.175894    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:37:25.177129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.177103    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:37:25.177352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.177317    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:37:25.177352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.177317    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:37:25.180816 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.180793    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28" pod "0ceeffa5-00cc-4a12-8da7-fd463083e22a" (UID: "0ceeffa5-00cc-4a12-8da7-fd463083e22a"). InnerVolumeSpecName "kube-api-access-dsd28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277540    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277572    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277583    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277591    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277604    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsd28\" (UniqueName: \"kubernetes.io/projected/0ceeffa5-00cc-4a12-8da7-fd463083e22a-kube-api-access-dsd28\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277616    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:25.277633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:25.277630    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ceeffa5-00cc-4a12-8da7-fd463083e22a-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:26.065839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.065801    4955 generic.go:356] "Generic (PLEG): container finished" podID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerID="19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021" exitCode=0
Nov 03 15:37:26.066387 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.065871    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" event={"ID":"168c6154-e58d-4ca2-86e3-4893f1b7d9ae","Type":"ContainerDied","Data":"19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"}
Nov 03 15:37:26.066387 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.066043    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"
Nov 03 15:37:26.066665 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.066649    4955 scope.go:122] "RemoveContainer" containerID="19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"
Nov 03 15:37:26.072697 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.072664    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:26.072802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.072739    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:26.072802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.072773    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:37:26.292317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.292276    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"]
Nov 03 15:37:26.336746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:26.336657    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-qc96z"]
Nov 03 15:37:27.075002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:27.074940    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" event={"ID":"168c6154-e58d-4ca2-86e3-4893f1b7d9ae","Type":"ContainerStarted","Data":"b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28"}
Nov 03 15:37:27.208416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:27.208378    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" path="/var/lib/kubelet/pods/0ceeffa5-00cc-4a12-8da7-fd463083e22a/volumes"
Nov 03 15:37:27.568582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:27.568544    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:37:27.568799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:27.568772    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 15:37:27.568882 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:27.568833    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 15:37:27.969318 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:37:27.969280    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:37:28.568834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:28.568797    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 15:37:28.569400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:28.568866    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 15:37:29.568654 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:29.568617    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 15:37:29.568845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:29.568678    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 15:37:30.569914 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:30.569871    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 15:37:30.570375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:30.569937    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 15:37:31.569435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:31.569398    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 15:37:31.569674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:31.569472    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 15:37:32.568961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:32.568911    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 15:37:32.569412 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:32.569029    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 15:37:33.573102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:33.573071    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:37:33.573572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:33.573557    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:37:33.574399 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:33.574376    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:37:34.051161 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:34.051128    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:37:35.060145 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:35.060113    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:37:42.316929 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.316885    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"]
Nov 03 15:37:42.318581 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.318552    4955 cpu_manager.go:414] "containerMap: removing container" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy"
Nov 03 15:37:42.318581 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.318580    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy"
Nov 03 15:37:42.318807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.318779    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ceeffa5-00cc-4a12-8da7-fd463083e22a" containerName="istio-proxy"
Nov 03 15:37:42.323048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.323021    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.380189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.380158    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"]
Nov 03 15:37:42.426434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.426403    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5cad072f\"/\"kube-root-ca.crt\""
Nov 03 15:37:42.426612 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.426401    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5cad072f\"/\"openshift-service-ca.crt\""
Nov 03 15:37:42.431171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.431147    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5cad072f\"/\"default-dockercfg-7jl6l\""
Nov 03 15:37:42.431307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.431233    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5cad072f\"/\"e2e-lifecycle-model-5cad072f-kserve-self-signed-certs\""
Nov 03 15:37:42.479741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.477059    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.479741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.477174    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.479741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.477296    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c555\" (UniqueName: \"kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.479741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.477366    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.479741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.477440    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.479741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.477472    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.579448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.579365    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.579648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.579459    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c555\" (UniqueName: \"kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.579648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.579504    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.579648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.579551    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.579648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.579571    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.579648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.579644    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.580048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.580028    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.580048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.580037    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.580164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.580076    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.582093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.582069    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.582274 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.582258    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.669545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.669510    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c555\" (UniqueName: \"kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555\") pod \"e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") " pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:42.938409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:42.938372    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:43.344087 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:43.344050    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"]
Nov 03 15:37:43.345151 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:37:43.345117    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e67cca_1021_487e_9c5a_8789af0a3426.slice/crio-4622410b89361ae8cb29170f1d689820f6fdfec8b32c30f5de22402683e341c6 WatchSource:0}: Error finding container 4622410b89361ae8cb29170f1d689820f6fdfec8b32c30f5de22402683e341c6: Status 404 returned error can't find the container with id 4622410b89361ae8cb29170f1d689820f6fdfec8b32c30f5de22402683e341c6
Nov 03 15:37:43.382487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:43.382439    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" event={"ID":"82e67cca-1021-487e-9c5a-8789af0a3426","Type":"ContainerStarted","Data":"4622410b89361ae8cb29170f1d689820f6fdfec8b32c30f5de22402683e341c6"}
Nov 03 15:37:44.390673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:44.390631    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" event={"ID":"82e67cca-1021-487e-9c5a-8789af0a3426","Type":"ContainerStarted","Data":"2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96"}
Nov 03 15:37:44.510130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:44.510068    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" podStartSLOduration=2.510048883 podStartE2EDuration="2.510048883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:37:44.509227241 +0000 UTC m=+1779.434045154" watchObservedRunningTime="2026-11-03 15:37:44.510048883 +0000 UTC m=+1779.434866799"
Nov 03 15:37:44.939068 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:44.939032    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:49.072301 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:49.072262    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"]
Nov 03 15:37:49.192516 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:49.192487    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"]
Nov 03 15:37:52.939572 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:52.939540    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:52.952560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:52.952532    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:53.464872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:53.464829    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:37:54.073267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.073220    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx" podUID="2b4f4d06-c595-46c2-88f6-1fe037fbff20" containerName="payload-processing" containerID="cri-o://a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f" gracePeriod=25
Nov 03 15:37:54.193863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.193782    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp" podUID="5cef74ee-4103-48b7-9d2d-0530b73d24e1" containerName="payload-pre-processing" containerID="cri-o://4384e74ed0719693204a0e2eb73bc299e8c7a91f25bfa431fa6b00b6f7de040a" gracePeriod=25
Nov 03 15:37:54.360575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.360549    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:37:54.465112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.465077    4955 generic.go:356] "Generic (PLEG): container finished" podID="5cef74ee-4103-48b7-9d2d-0530b73d24e1" containerID="4384e74ed0719693204a0e2eb73bc299e8c7a91f25bfa431fa6b00b6f7de040a" exitCode=0
Nov 03 15:37:54.465271 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.465253    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp" event={"ID":"5cef74ee-4103-48b7-9d2d-0530b73d24e1","Type":"ContainerDied","Data":"4384e74ed0719693204a0e2eb73bc299e8c7a91f25bfa431fa6b00b6f7de040a"}
Nov 03 15:37:54.465317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.465309    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp" event={"ID":"5cef74ee-4103-48b7-9d2d-0530b73d24e1","Type":"ContainerDied","Data":"46709fb5512c2a9e954a943ab0a6ac66420463d0df59ec5a45843365c23d374c"}
Nov 03 15:37:54.465356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.465331    4955 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46709fb5512c2a9e954a943ab0a6ac66420463d0df59ec5a45843365c23d374c"
Nov 03 15:37:54.467023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.466982    4955 generic.go:356] "Generic (PLEG): container finished" podID="2b4f4d06-c595-46c2-88f6-1fe037fbff20" containerID="a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f" exitCode=0
Nov 03 15:37:54.467158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.467025    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx" event={"ID":"2b4f4d06-c595-46c2-88f6-1fe037fbff20","Type":"ContainerDied","Data":"a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f"}
Nov 03 15:37:54.467158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.467058    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"
Nov 03 15:37:54.467158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.467078    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx" event={"ID":"2b4f4d06-c595-46c2-88f6-1fe037fbff20","Type":"ContainerDied","Data":"74b34bddf2231ebf2955bfb194a1d8bbc21155ea367da0d396a6e407babd5df1"}
Nov 03 15:37:54.467158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.467101    4955 scope.go:122] "RemoveContainer" containerID="a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f"
Nov 03 15:37:54.477158 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.477135    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:37:54.481317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.481292    4955 scope.go:122] "RemoveContainer" containerID="a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f"
Nov 03 15:37:54.481719 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:37:54.481688    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f\": container with ID starting with a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f not found: ID does not exist" containerID="a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f"
Nov 03 15:37:54.481858 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.481732    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f"} err="failed to get container status \"a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f\": rpc error: code = NotFound desc = could not find container \"a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f\": container with ID starting with a6cfc21018d3dabeabf4f62605868750e6b729ef75ef1f5e8033c7303e39e77f not found: ID does not exist"
Nov 03 15:37:54.490317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.490291    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume\") pod \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") "
Nov 03 15:37:54.490556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.490537    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd\" (UniqueName: \"kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd\") pod \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\" (UID: \"2b4f4d06-c595-46c2-88f6-1fe037fbff20\") "
Nov 03 15:37:54.490733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.490707    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume" pod "2b4f4d06-c595-46c2-88f6-1fe037fbff20" (UID: "2b4f4d06-c595-46c2-88f6-1fe037fbff20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:37:54.491125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.491097    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f4d06-c595-46c2-88f6-1fe037fbff20-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:54.493264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.493239    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd" pod "2b4f4d06-c595-46c2-88f6-1fe037fbff20" (UID: "2b4f4d06-c595-46c2-88f6-1fe037fbff20"). InnerVolumeSpecName "kube-api-access-f4gvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:37:54.591782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.591685    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck\" (UniqueName: \"kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck\") pod \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") "
Nov 03 15:37:54.592008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.591953    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume\") pod \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\" (UID: \"5cef74ee-4103-48b7-9d2d-0530b73d24e1\") "
Nov 03 15:37:54.592319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.592280    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume" pod "5cef74ee-4103-48b7-9d2d-0530b73d24e1" (UID: "5cef74ee-4103-48b7-9d2d-0530b73d24e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:37:54.592609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.592490    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cef74ee-4103-48b7-9d2d-0530b73d24e1-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:54.592609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.592506    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gvd\" (UniqueName: \"kubernetes.io/projected/2b4f4d06-c595-46c2-88f6-1fe037fbff20-kube-api-access-f4gvd\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:54.594025 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.594001    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck" pod "5cef74ee-4103-48b7-9d2d-0530b73d24e1" (UID: "5cef74ee-4103-48b7-9d2d-0530b73d24e1"). InnerVolumeSpecName "kube-api-access-g62ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:37:54.694204 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.694163    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g62ck\" (UniqueName: \"kubernetes.io/projected/5cef74ee-4103-48b7-9d2d-0530b73d24e1-kube-api-access-g62ck\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:37:54.891434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.891404    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"]
Nov 03 15:37:54.897620 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:54.897589    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-42595854-7cd4b47b7f-kmhkx"]
Nov 03 15:37:55.208611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:55.208536    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4f4d06-c595-46c2-88f6-1fe037fbff20" path="/var/lib/kubelet/pods/2b4f4d06-c595-46c2-88f6-1fe037fbff20/volumes"
Nov 03 15:37:55.476384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:55.476307    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"
Nov 03 15:37:55.643782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:55.643745    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"]
Nov 03 15:37:55.647852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:55.647820    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-42595854-54b857b995-fqscp"]
Nov 03 15:37:57.207771 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:37:57.207737    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cef74ee-4103-48b7-9d2d-0530b73d24e1" path="/var/lib/kubelet/pods/5cef74ee-4103-48b7-9d2d-0530b73d24e1/volumes"
Nov 03 15:37:57.940731 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:37:57.940695    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:38:04.420981 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.420938    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"]
Nov 03 15:38:04.421649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.421631    4955 cpu_manager.go:414] "containerMap: removing container" podUID="5cef74ee-4103-48b7-9d2d-0530b73d24e1" containerName="payload-pre-processing"
Nov 03 15:38:04.421703 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.421649    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cef74ee-4103-48b7-9d2d-0530b73d24e1" containerName="payload-pre-processing"
Nov 03 15:38:04.421703 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.421671    4955 cpu_manager.go:414] "containerMap: removing container" podUID="2b4f4d06-c595-46c2-88f6-1fe037fbff20" containerName="payload-processing"
Nov 03 15:38:04.421703 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.421675    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4f4d06-c595-46c2-88f6-1fe037fbff20" containerName="payload-processing"
Nov 03 15:38:04.421820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.421781    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4f4d06-c595-46c2-88f6-1fe037fbff20" containerName="payload-processing"
Nov 03 15:38:04.421820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.421797    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cef74ee-4103-48b7-9d2d-0530b73d24e1" containerName="payload-pre-processing"
Nov 03 15:38:04.426332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.426307    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.480626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.480597    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-f356bb-openshift-default-dockercfg-vfkq5\""
Nov 03 15:38:04.536015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.535599    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"]
Nov 03 15:38:04.594672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594630    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.594859 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594679    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.594859 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594725    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.594859 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594773    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtd2b\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.594859 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594851    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.595102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594896    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.595102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.594918    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.595212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.595107    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.595289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.595273    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703289    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703376    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703496    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703613    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703648    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703680    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703711    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtd2b\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703774    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.704047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.703844    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.731991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.714469    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.731991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.715116    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.731991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.725232    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.731991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.725526    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.731991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.725754    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.750065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.748272    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.753751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.753635    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.783738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.783641    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtd2b\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:04.784319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:04.784290    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token\") pod \"e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") " pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:05.041102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.041014    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:05.481836 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:38:05.481798    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb578c022_7306_43d9_8886_4f16c09db6c1.slice/crio-64b303cf2c02f5788d91d4edce6326489899fc77e4fcded1f113f358e1f3cc36 WatchSource:0}: Error finding container 64b303cf2c02f5788d91d4edce6326489899fc77e4fcded1f113f358e1f3cc36: Status 404 returned error can't find the container with id 64b303cf2c02f5788d91d4edce6326489899fc77e4fcded1f113f358e1f3cc36
Nov 03 15:38:05.482290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.482209    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"]
Nov 03 15:38:05.484819 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.484782    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:05.484896 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.484853    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:05.484896 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.484882    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:05.546450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.546418    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" event={"ID":"b578c022-7306-43d9-8886-4f16c09db6c1","Type":"ContainerStarted","Data":"64b303cf2c02f5788d91d4edce6326489899fc77e4fcded1f113f358e1f3cc36"}
Nov 03 15:38:05.707152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.707118    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:38:05.715656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.715631    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:38:05.718139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.718119    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:38:05.727326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.727289    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:38:05.784841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:05.784762    4955 scope.go:122] "RemoveContainer" containerID="4d0e4fec4598de0bdfbeb48aa45ef884f50f179238353f2dd5855b9917bcafe9"
Nov 03 15:38:06.511108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:06.511075    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"]
Nov 03 15:38:06.511546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:06.511270    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z" podUID="1e71cdc8-f7ca-4f55-999d-4487a2648e3b" containerName="maas-api" containerID="cri-o://0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1" gracePeriod=30
Nov 03 15:38:06.555109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:06.555075    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" event={"ID":"b578c022-7306-43d9-8886-4f16c09db6c1","Type":"ContainerStarted","Data":"4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"}
Nov 03 15:38:06.763041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:06.762893    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podStartSLOduration=2.762872121 podStartE2EDuration="2.762872121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:06.758564606 +0000 UTC m=+1801.683382518" watchObservedRunningTime="2026-11-03 15:38:06.762872121 +0000 UTC m=+1801.687690035"
Nov 03 15:38:06.915930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:06.915903    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:38:07.037408 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.037324    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9\" (UniqueName: \"kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9\") pod \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") "
Nov 03 15:38:07.037590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.037495    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\") pod \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") "
Nov 03 15:38:07.037590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.037556    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs\") pod \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") "
Nov 03 15:38:07.037725 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.037699    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca\") pod \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\" (UID: \"1e71cdc8-f7ca-4f55-999d-4487a2648e3b\") "
Nov 03 15:38:07.038422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.038261    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca" pod "1e71cdc8-f7ca-4f55-999d-4487a2648e3b" (UID: "1e71cdc8-f7ca-4f55-999d-4487a2648e3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:38:07.040113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.040071    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls" pod "1e71cdc8-f7ca-4f55-999d-4487a2648e3b" (UID: "1e71cdc8-f7ca-4f55-999d-4487a2648e3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:38:07.040212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.040174    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9" pod "1e71cdc8-f7ca-4f55-999d-4487a2648e3b" (UID: "1e71cdc8-f7ca-4f55-999d-4487a2648e3b"). InnerVolumeSpecName "kube-api-access-qvzh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:07.040212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.040184    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs" pod "1e71cdc8-f7ca-4f55-999d-4487a2648e3b" (UID: "1e71cdc8-f7ca-4f55-999d-4487a2648e3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:38:07.041304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.041282    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:07.049097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.049071    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:07.140100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.140063    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvzh9\" (UniqueName: \"kubernetes.io/projected/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-kube-api-access-qvzh9\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:07.140100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.140094    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:07.140100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.140104    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:07.140342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.140113    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e71cdc8-f7ca-4f55-999d-4487a2648e3b-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:07.630086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.627674    4955 generic.go:356] "Generic (PLEG): container finished" podID="1e71cdc8-f7ca-4f55-999d-4487a2648e3b" containerID="0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1" exitCode=0
Nov 03 15:38:07.652996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.648104    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"
Nov 03 15:38:07.652996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.648943    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z" event={"ID":"1e71cdc8-f7ca-4f55-999d-4487a2648e3b","Type":"ContainerDied","Data":"0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1"}
Nov 03 15:38:07.652996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.649024    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z" event={"ID":"1e71cdc8-f7ca-4f55-999d-4487a2648e3b","Type":"ContainerDied","Data":"2d4c7fed65baba875e628e243856cba8aeecda491dc5b4930b1b670b7fbd5bbb"}
Nov 03 15:38:07.652996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.649056    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:07.652996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.649088    4955 scope.go:122] "RemoveContainer" containerID="0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1"
Nov 03 15:38:07.665809 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.665453    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:38:07.697240 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.696727    4955 scope.go:122] "RemoveContainer" containerID="0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1"
Nov 03 15:38:07.697648 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:07.697551    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1\": container with ID starting with 0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1 not found: ID does not exist" containerID="0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1"
Nov 03 15:38:07.697648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.697588    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1"} err="failed to get container status \"0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1\": rpc error: code = NotFound desc = could not find container \"0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1\": container with ID starting with 0ac6eff7977c63c93f04431977c90274aab663852a3f9e56b833777cf16a50d1 not found: ID does not exist"
Nov 03 15:38:07.958360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.958321    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"]
Nov 03 15:38:07.962223 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:07.962188    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-42595854-6d8dd475db-fct9z"]
Nov 03 15:38:09.012468 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.012434    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"]
Nov 03 15:38:09.013348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.013327    4955 cpu_manager.go:414] "containerMap: removing container" podUID="1e71cdc8-f7ca-4f55-999d-4487a2648e3b" containerName="maas-api"
Nov 03 15:38:09.013471 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.013350    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e71cdc8-f7ca-4f55-999d-4487a2648e3b" containerName="maas-api"
Nov 03 15:38:09.013543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.013528    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e71cdc8-f7ca-4f55-999d-4487a2648e3b" containerName="maas-api"
Nov 03 15:38:09.018395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.018372    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.079897 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.079866    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"]
Nov 03 15:38:09.106898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.106857    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.107113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.106956    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.107652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.107166    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.107652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.107242    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.107652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.107298    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.107652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.107327    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nww2c\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.108157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.108068    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.108281 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.108202    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.108281 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.108237    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.124185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.124162    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-771ea0-openshift-default-dockercfg-t6nbf\""
Nov 03 15:38:09.209102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.209070    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e71cdc8-f7ca-4f55-999d-4487a2648e3b" path="/var/lib/kubelet/pods/1e71cdc8-f7ca-4f55-999d-4487a2648e3b/volumes"
Nov 03 15:38:09.210634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.210588    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.210787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.210652    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.210787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.210702    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.210787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.210731    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nww2c\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.210787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.210762    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.210925    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211059 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211029    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211121    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211363 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211342    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211472    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211592    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211696 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211642    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.211696 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.211671    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.212143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.212122    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.213429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.213404    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.213775 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.213756    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.325903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.325823    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nww2c\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.354626 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.354586    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token\") pod \"e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") " pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:09.634864 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:09.634786    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:10.079135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.079103    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"]
Nov 03 15:38:10.079581 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:38:10.079328    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab2ea66_3da5_4b85_979f_a40a2253b523.slice/crio-ef16e2e1f663bece8c07b217f0b6603fee9d8a265d5086890b8595ca8293e0d4 WatchSource:0}: Error finding container ef16e2e1f663bece8c07b217f0b6603fee9d8a265d5086890b8595ca8293e0d4: Status 404 returned error can't find the container with id ef16e2e1f663bece8c07b217f0b6603fee9d8a265d5086890b8595ca8293e0d4
Nov 03 15:38:10.082440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.082406    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:10.082509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.082475    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:10.082546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.082507    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:10.672926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.672874    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" event={"ID":"2ab2ea66-3da5-4b85-979f-a40a2253b523","Type":"ContainerStarted","Data":"66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"}
Nov 03 15:38:10.673181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.673030    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" event={"ID":"2ab2ea66-3da5-4b85-979f-a40a2253b523","Type":"ContainerStarted","Data":"ef16e2e1f663bece8c07b217f0b6603fee9d8a265d5086890b8595ca8293e0d4"}
Nov 03 15:38:10.799364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:10.799306    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podStartSLOduration=2.799289595 podStartE2EDuration="2.799289595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:10.796059288 +0000 UTC m=+1805.720877202" watchObservedRunningTime="2026-11-03 15:38:10.799289595 +0000 UTC m=+1805.724107507"
Nov 03 15:38:11.536198 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:11.536161    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:38:11.635598 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:11.635562    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:11.641380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:11.641354    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:11.679751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:11.679719    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:11.680810 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:11.680785    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:38:18.084520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:18.084482    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"]
Nov 03 15:38:18.085136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:18.084754    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy" containerID="cri-o://bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2" gracePeriod=30
Nov 03 15:38:19.303898 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.303863    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"]
Nov 03 15:38:19.323850 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.323816    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.377154 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.377124    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-fe0cb9-serving-cert\""
Nov 03 15:38:19.432739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.432705    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"]
Nov 03 15:38:19.478709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.478672    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.478941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.478918    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.479118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.479098    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.479199 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.479175    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhgj\" (UniqueName: \"kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.580872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.580781    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.580872 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.580855    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhgj\" (UniqueName: \"kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.581130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.580893    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.581130 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:19.580949    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-serving-cert: secret "maas-api-e2e-shared-a-w0-fe0cb9-serving-cert" not found
Nov 03 15:38:19.581130 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:19.581059    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls podName:8f46fe3a-c661-4ce1-8a74-f80c501c3c2e nodeName:}" failed. No retries permitted until 2026-11-03 15:38:20.081038012 +0000 UTC m=+1815.005855917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls") pod "maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" (UID: "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e") : secret "maas-api-e2e-shared-a-w0-fe0cb9-serving-cert" not found
Nov 03 15:38:19.581281 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.581149    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.581603 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.581582    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.583923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.583900    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:19.649907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:19.649877    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhgj\" (UniqueName: \"kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:20.144140 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:20.144106    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:20.147004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:20.146952    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:20.238662 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:20.238632    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:20.624452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:20.624414    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"]
Nov 03 15:38:20.628571 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:38:20.628539    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f46fe3a_c661_4ce1_8a74_f80c501c3c2e.slice/crio-2636e6c0942c2c2e33656b8cc31f0eb885244248b807cb29795ac6bf44c74b63 WatchSource:0}: Error finding container 2636e6c0942c2c2e33656b8cc31f0eb885244248b807cb29795ac6bf44c74b63: Status 404 returned error can't find the container with id 2636e6c0942c2c2e33656b8cc31f0eb885244248b807cb29795ac6bf44c74b63
Nov 03 15:38:20.816217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:20.816180    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" event={"ID":"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e","Type":"ContainerStarted","Data":"2636e6c0942c2c2e33656b8cc31f0eb885244248b807cb29795ac6bf44c74b63"}
Nov 03 15:38:21.824265 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:21.824184    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" event={"ID":"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e","Type":"ContainerStarted","Data":"b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc"}
Nov 03 15:38:21.824670 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:21.824322    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:21.967501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:21.967431    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" podStartSLOduration=2.127038217 podStartE2EDuration="2.967410153s" totalImagesPullingTime="840.371936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:21.962110013 +0000 UTC m=+1816.886927928" watchObservedRunningTime="2026-11-03 15:38:21.967410153 +0000 UTC m=+1816.892228081"
Nov 03 15:38:23.385425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.385396    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:38:23.440217 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440178    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440264    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440374    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440404    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440468    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440499    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440527    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440643    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440725    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440742    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440775    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.440779    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441071    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441150    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo\") pod \"99202fd8-8caa-48a7-9a5f-df882c1098eb\" (UID: \"99202fd8-8caa-48a7-9a5f-df882c1098eb\") "
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441787    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441807    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99202fd8-8caa-48a7-9a5f-df882c1098eb-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441820    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441834    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.442589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.441848    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.443610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.443579    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:23.443789 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.443638    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:23.444101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.444078    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "kube-api-access-qfdb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:23.446084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.446059    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo" pod "99202fd8-8caa-48a7-9a5f-df882c1098eb" (UID: "99202fd8-8caa-48a7-9a5f-df882c1098eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:38:23.543768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.543738    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.543768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.543767    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdb8\" (UniqueName: \"kubernetes.io/projected/99202fd8-8caa-48a7-9a5f-df882c1098eb-kube-api-access-qfdb8\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.543961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.543778    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.543961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.543787    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99202fd8-8caa-48a7-9a5f-df882c1098eb-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:23.683069 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.682983    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"]
Nov 03 15:38:23.856127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.856048    4955 generic.go:356] "Generic (PLEG): container finished" podID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerID="c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616" exitCode=0
Nov 03 15:38:23.856537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.856161    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" event={"ID":"691ab01b-5d2e-4dd9-868e-c49f850e2e67","Type":"ContainerDied","Data":"c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"}
Nov 03 15:38:23.856716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.856683    4955 scope.go:122] "RemoveContainer" containerID="c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"
Nov 03 15:38:23.880042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.859821    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:23.880042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.859929    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:23.880042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.859998    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:23.919221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.904526    4955 generic.go:356] "Generic (PLEG): container finished" podID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerID="bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2" exitCode=0
Nov 03 15:38:23.919221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.904841    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" event={"ID":"99202fd8-8caa-48a7-9a5f-df882c1098eb","Type":"ContainerDied","Data":"bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2"}
Nov 03 15:38:23.919221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.904931    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts" event={"ID":"99202fd8-8caa-48a7-9a5f-df882c1098eb","Type":"ContainerDied","Data":"0237fb6c941f01b2a74018436bf4eac591513158758ef5e19aff54991c9e242f"}
Nov 03 15:38:23.919221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.905002    4955 scope.go:122] "RemoveContainer" containerID="bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2"
Nov 03 15:38:23.919221 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.905170    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"
Nov 03 15:38:23.964205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.964154    4955 scope.go:122] "RemoveContainer" containerID="9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"
Nov 03 15:38:23.974541 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.974520    4955 scope.go:122] "RemoveContainer" containerID="bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2"
Nov 03 15:38:23.974851 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:23.974832    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2\": container with ID starting with bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2 not found: ID does not exist" containerID="bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2"
Nov 03 15:38:23.974913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.974861    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2"} err="failed to get container status \"bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2\": rpc error: code = NotFound desc = could not find container \"bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2\": container with ID starting with bc2cb64dfdca3fa65cf262e28880640162384f8ac8cccd444281c4d51e600ae2 not found: ID does not exist"
Nov 03 15:38:23.974913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.974882    4955 scope.go:122] "RemoveContainer" containerID="9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"
Nov 03 15:38:23.975145 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:23.975121    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab\": container with ID starting with 9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab not found: ID does not exist" containerID="9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"
Nov 03 15:38:23.975188 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:23.975153    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab"} err="failed to get container status \"9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab\": rpc error: code = NotFound desc = could not find container \"9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab\": container with ID starting with 9e368351453f76ae2efcdf666cc857a11788925072fc2a8aa11ff84d6bfceaab not found: ID does not exist"
Nov 03 15:38:24.138081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:24.138051    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"]
Nov 03 15:38:24.141547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:24.141511    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-42595854-openshift-default-8476dc6c85-txfts"]
Nov 03 15:38:24.914195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:24.914142    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" event={"ID":"691ab01b-5d2e-4dd9-868e-c49f850e2e67","Type":"ContainerStarted","Data":"0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf"}
Nov 03 15:38:25.208257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:25.208163    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" path="/var/lib/kubelet/pods/99202fd8-8caa-48a7-9a5f-df882c1098eb/volumes"
Nov 03 15:38:25.458049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:25.458011    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:38:25.458631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:25.458564    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 15:38:25.458763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:25.458653    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 15:38:26.458775 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:26.458734    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 15:38:26.459231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:26.458800    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 15:38:27.459014 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:27.458945    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 15:38:27.459456 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:27.459041    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 15:38:27.860584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:27.860505    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:38:27.989645 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:27.989502    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:38:28.458800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:28.458759    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 15:38:28.459024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:28.458834    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 15:38:29.458869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:29.458833    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 15:38:29.459347 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:29.458889    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 15:38:30.463653 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:30.463622    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:38:30.464097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:30.464075    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:38:30.464861 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:30.464842    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:38:32.886071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.886039    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"]
Nov 03 15:38:32.886935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.886913    4955 cpu_manager.go:414] "containerMap: removing container" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy"
Nov 03 15:38:32.886935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.886934    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy"
Nov 03 15:38:32.887147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.887102    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy"
Nov 03 15:38:32.887147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.887119    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy"
Nov 03 15:38:32.892394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.892374    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:32.954565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:32.954535    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"]
Nov 03 15:38:33.000170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.000127    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5cad072f-openshift-default-dockercfg-2j2g5\""
Nov 03 15:38:33.018298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018263    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018341    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018413    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018451    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018531    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018570    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018673    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lqqb\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018744    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.018906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.018893    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120579    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120624    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120648    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120665    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120687    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120710    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.120792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120751    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lqqb\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120796    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.120963    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.121231    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121333 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.121305    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.121385    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.121493    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.121734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.121613    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.123264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.123241    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.123487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.123465    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.184309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.184280    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lqqb\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.184465 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.184415    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.207427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.207395    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:33.661178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.661134    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"]
Nov 03 15:38:33.672505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.671472    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:33.673036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.672777    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:33.673036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:33.672874    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:34.031871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:34.031829    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" event={"ID":"06a2dec3-269a-4bd3-831c-b1be9dfa8359","Type":"ContainerStarted","Data":"914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863"}
Nov 03 15:38:34.032308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:34.031891    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" event={"ID":"06a2dec3-269a-4bd3-831c-b1be9dfa8359","Type":"ContainerStarted","Data":"e44f905bc049eecd09e06f43c16cc3d84e413acbaa38c7691253b8cc1f5847e2"}
Nov 03 15:38:34.154848 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:34.154797    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podStartSLOduration=2.154779076 podStartE2EDuration="2.154779076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:38:34.153432644 +0000 UTC m=+1829.078250558" watchObservedRunningTime="2026-11-03 15:38:34.154779076 +0000 UTC m=+1829.079596995"
Nov 03 15:38:34.208422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:34.208336    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:34.209954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:34.209929    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:34.210089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:34.209989    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:35.208040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:35.208007    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:35.208436 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:35.208056    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:36.207841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:36.207804    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:36.208036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:36.207853    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:37.209944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:37.209842    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:37.209944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:37.209932    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:38.211003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:38.209333    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:38.211003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:38.209405    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:38.705761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:38.705719    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main" containerID="cri-o://2ca1a24980d2fd2d3d861a24ccb3f1c7e8ab7f15594586c9d8c3619319be4f02" gracePeriod=45
Nov 03 15:38:39.213002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.210317    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:39.213002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.210386    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:39.295165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.295122    4955 generic.go:356] "Generic (PLEG): container finished" podID="05297afe-2652-4157-8383-bcb8fc7c2851" containerID="2ca1a24980d2fd2d3d861a24ccb3f1c7e8ab7f15594586c9d8c3619319be4f02" exitCode=0
Nov 03 15:38:39.300113 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.300079    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" event={"ID":"05297afe-2652-4157-8383-bcb8fc7c2851","Type":"ContainerDied","Data":"2ca1a24980d2fd2d3d861a24ccb3f1c7e8ab7f15594586c9d8c3619319be4f02"}
Nov 03 15:38:39.337005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.331646    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:38:39.414131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414096    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm\") pod \"05297afe-2652-4157-8383-bcb8fc7c2851\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") "
Nov 03 15:38:39.414313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414158    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home\") pod \"05297afe-2652-4157-8383-bcb8fc7c2851\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") "
Nov 03 15:38:39.414313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414251    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs\" (UniqueName: \"kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs\") pod \"05297afe-2652-4157-8383-bcb8fc7c2851\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") "
Nov 03 15:38:39.414436 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414387    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m\" (UniqueName: \"kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m\") pod \"05297afe-2652-4157-8383-bcb8fc7c2851\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") "
Nov 03 15:38:39.414495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414451    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home" pod "05297afe-2652-4157-8383-bcb8fc7c2851" (UID: "05297afe-2652-4157-8383-bcb8fc7c2851"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:39.414495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414470    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir\") pod \"05297afe-2652-4157-8383-bcb8fc7c2851\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") "
Nov 03 15:38:39.414611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414534    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache\") pod \"05297afe-2652-4157-8383-bcb8fc7c2851\" (UID: \"05297afe-2652-4157-8383-bcb8fc7c2851\") "
Nov 03 15:38:39.414764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414738    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir" pod "05297afe-2652-4157-8383-bcb8fc7c2851" (UID: "05297afe-2652-4157-8383-bcb8fc7c2851"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:39.414853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.414813    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache" pod "05297afe-2652-4157-8383-bcb8fc7c2851" (UID: "05297afe-2652-4157-8383-bcb8fc7c2851"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:39.415255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.415235    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:39.415255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.415255    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:39.415444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.415265    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:39.416630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.416601    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm" pod "05297afe-2652-4157-8383-bcb8fc7c2851" (UID: "05297afe-2652-4157-8383-bcb8fc7c2851"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:39.416726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.416690    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs" pod "05297afe-2652-4157-8383-bcb8fc7c2851" (UID: "05297afe-2652-4157-8383-bcb8fc7c2851"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:38:39.416766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.416727    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m" pod "05297afe-2652-4157-8383-bcb8fc7c2851" (UID: "05297afe-2652-4157-8383-bcb8fc7c2851"). InnerVolumeSpecName "kube-api-access-zn49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:39.533107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.532179    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn49m\" (UniqueName: \"kubernetes.io/projected/05297afe-2652-4157-8383-bcb8fc7c2851-kube-api-access-zn49m\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:39.533107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.532221    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05297afe-2652-4157-8383-bcb8fc7c2851-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:39.533107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:39.532240    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05297afe-2652-4157-8383-bcb8fc7c2851-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:40.209693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.209657    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 15:38:40.209879 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.209708    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 15:38:40.304232 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.304205    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"
Nov 03 15:38:40.304632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.304241    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx" event={"ID":"05297afe-2652-4157-8383-bcb8fc7c2851","Type":"ContainerDied","Data":"ab3f43afe7fbbd7cf918f51d9efbbd2617a3381c063e44af94f430a12126285a"}
Nov 03 15:38:40.304632 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.304288    4955 scope.go:122] "RemoveContainer" containerID="2ca1a24980d2fd2d3d861a24ccb3f1c7e8ab7f15594586c9d8c3619319be4f02"
Nov 03 15:38:40.477422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.477335    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"]
Nov 03 15:38:40.481333 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:40.481309    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f78fd6/distinct-w2-f78fd6-kserve-679cb9965b-5qbzx"]
Nov 03 15:38:41.207357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:41.207321    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" path="/var/lib/kubelet/pods/05297afe-2652-4157-8383-bcb8fc7c2851/volumes"
Nov 03 15:38:41.212345 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:41.212321    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:41.212769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:41.212752    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:41.213593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:41.213575    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:38:50.408005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.407934    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"]
Nov 03 15:38:50.408560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.408174    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy" containerID="cri-o://f4f192f2b58ef55472cba93d44b87f5a530a96c4c5eccc4f4b7defaf8eb66494" gracePeriod=30
Nov 03 15:38:50.423300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.419714    4955 generic.go:356] "Generic (PLEG): container finished" podID="4ba150f8-5538-4677-90c9-178795e8c449" containerID="d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051" exitCode=0
Nov 03 15:38:50.423300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.419986    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" event={"ID":"4ba150f8-5538-4677-90c9-178795e8c449","Type":"ContainerDied","Data":"d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"}
Nov 03 15:38:50.423300 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.420516    4955 scope.go:122] "RemoveContainer" containerID="d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"
Nov 03 15:38:50.425364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.425320    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:50.425501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.425423    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:50.425501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:50.425472    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:38:51.428822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:51.428783    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" event={"ID":"4ba150f8-5538-4677-90c9-178795e8c449","Type":"ContainerStarted","Data":"81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0"}
Nov 03 15:38:51.459838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:51.459801    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"]
Nov 03 15:38:51.460087 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:51.460062    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy" containerID="cri-o://5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf" gracePeriod=30
Nov 03 15:38:51.569006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:51.568146    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:38:51.569006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:51.568280    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 15:38:51.569006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:51.568323    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 15:38:52.566557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:52.566520    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 15:38:52.567002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:52.566576    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 15:38:53.566858 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:53.566819    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 15:38:53.567337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:53.566871    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 15:38:54.575998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:54.567072    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 15:38:54.575998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:54.567132    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 15:38:55.562329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.562293    4955 generic.go:356] "Generic (PLEG): container finished" podID="cb14684e-4893-48b6-9c94-ffa657288564" containerID="f4f192f2b58ef55472cba93d44b87f5a530a96c4c5eccc4f4b7defaf8eb66494" exitCode=0
Nov 03 15:38:55.562554 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.562349    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" event={"ID":"cb14684e-4893-48b6-9c94-ffa657288564","Type":"ContainerDied","Data":"f4f192f2b58ef55472cba93d44b87f5a530a96c4c5eccc4f4b7defaf8eb66494"}
Nov 03 15:38:55.566348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.566325    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 15:38:55.566443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.566372    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 15:38:55.728557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.728529    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:38:55.803042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803005    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803081    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803205    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803241    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803341 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803334    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803384    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:55.803450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803423    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803429    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:55.803616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803486    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803656    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.803733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803686    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:55.803733 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803707    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:38:55.803849 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.803781    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn\") pod \"cb14684e-4893-48b6-9c94-ffa657288564\" (UID: \"cb14684e-4893-48b6-9c94-ffa657288564\") "
Nov 03 15:38:55.804263 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.804032    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:55.804503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.804484    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14684e-4893-48b6-9c94-ffa657288564-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.804605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.804507    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.804605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.804521    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.804605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.804534    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.804605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.804546    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.806481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.806452    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:55.806577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.806532    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:55.806729 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.806702    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "kube-api-access-5klxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:55.807409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.807384    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo" pod "cb14684e-4893-48b6-9c94-ffa657288564" (UID: "cb14684e-4893-48b6-9c94-ffa657288564"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:38:55.906286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.906250    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14684e-4893-48b6-9c94-ffa657288564-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.906286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.906282    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14684e-4893-48b6-9c94-ffa657288564-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.906499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.906296    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:55.906499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:55.906310    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5klxn\" (UniqueName: \"kubernetes.io/projected/cb14684e-4893-48b6-9c94-ffa657288564-kube-api-access-5klxn\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.567304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.566839    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 15:38:56.567304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.566895    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 15:38:56.578493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.578262    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"
Nov 03 15:38:56.578493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.578282    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x" event={"ID":"cb14684e-4893-48b6-9c94-ffa657288564","Type":"ContainerDied","Data":"1e419a7f7e763a192d2a769c572d84117fa97fab50b204ff4ece030dae2bbba1"}
Nov 03 15:38:56.578493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.578351    4955 scope.go:122] "RemoveContainer" containerID="f4f192f2b58ef55472cba93d44b87f5a530a96c4c5eccc4f4b7defaf8eb66494"
Nov 03 15:38:56.776135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.776093    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:38:56.783560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.783530    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"]
Nov 03 15:38:56.829072 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829036    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829081    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829119    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829139    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829253 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829191    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829329    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829388    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829509    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829558    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:56.829629 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829610    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:38:56.829670 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829637    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data\") pod \"76de34ed-49b4-405e-8161-978f3af6c0f2\" (UID: \"76de34ed-49b4-405e-8161-978f3af6c0f2\") "
Nov 03 15:38:56.829821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829799    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:56.829909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.829844    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:56.830127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.830103    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:56.830493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.830470    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.830758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.830499    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.830758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.830513    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.830758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.830526    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.830758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.830539    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76de34ed-49b4-405e-8161-978f3af6c0f2-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.832010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.831921    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-ndn8x"]
Nov 03 15:38:56.832313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.832288    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:38:56.832425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.832409    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "kube-api-access-rcf45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:56.832529 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.832509    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:38:56.832593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.832567    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token" pod "76de34ed-49b4-405e-8161-978f3af6c0f2" (UID: "76de34ed-49b4-405e-8161-978f3af6c0f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:38:56.932684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.932638    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcf45\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-kube-api-access-rcf45\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.932684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.932678    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.932955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.932710    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:56.932955 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:56.932726    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76de34ed-49b4-405e-8161-978f3af6c0f2-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:38:57.210774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.210736    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb14684e-4893-48b6-9c94-ffa657288564" path="/var/lib/kubelet/pods/cb14684e-4893-48b6-9c94-ffa657288564/volumes"
Nov 03 15:38:57.571722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.571650    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:38:57.572098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.572079    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:38:57.572839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.572818    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:38:57.587800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.587770    4955 generic.go:356] "Generic (PLEG): container finished" podID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerID="5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf" exitCode=0
Nov 03 15:38:57.587939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.587841    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"
Nov 03 15:38:57.587939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.587858    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" event={"ID":"76de34ed-49b4-405e-8161-978f3af6c0f2","Type":"ContainerDied","Data":"5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf"}
Nov 03 15:38:57.587939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.587918    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq" event={"ID":"76de34ed-49b4-405e-8161-978f3af6c0f2","Type":"ContainerDied","Data":"f55dc3174d57d0e6418ee7be239f6eedaccfc0e40ea7005b0f21e6a67839f926"}
Nov 03 15:38:57.588125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.587947    4955 scope.go:122] "RemoveContainer" containerID="5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf"
Nov 03 15:38:57.598485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.598467    4955 scope.go:122] "RemoveContainer" containerID="5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf"
Nov 03 15:38:57.598759 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:57.598742    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf\": container with ID starting with 5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf not found: ID does not exist" containerID="5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf"
Nov 03 15:38:57.598815 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.598768    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf"} err="failed to get container status \"5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf\": rpc error: code = NotFound desc = could not find container \"5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf\": container with ID starting with 5b529a5f97a8595dfd853de377768105f48029750a4feeac81e2bc3c27d46ccf not found: ID does not exist"
Nov 03 15:38:57.951377 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:38:57.951333    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:38:57.955279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.955250    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"]
Nov 03 15:38:57.958422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:57.958397    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-f9zrq"]
Nov 03 15:38:59.207735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:38:59.207702    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" path="/var/lib/kubelet/pods/76de34ed-49b4-405e-8161-978f3af6c0f2/volumes"
Nov 03 15:39:11.676605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:11.676570    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"]
Nov 03 15:39:11.783386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:11.783354    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"]
Nov 03 15:39:16.682122 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:16.682059    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r" podUID="5c1c4cfe-7955-43ec-939d-450c1333a38c" containerName="payload-processing" containerID="cri-o://20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43" gracePeriod=25
Nov 03 15:39:16.784008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:16.783956    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds" podUID="fb3e5f81-f567-4156-b6ac-981a1d962408" containerName="payload-pre-processing" containerID="cri-o://44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23" gracePeriod=25
Nov 03 15:39:17.149033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.149003    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:39:17.158526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.158498    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:39:17.215085 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.215060    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:39:17.218400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.218375    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6\" (UniqueName: \"kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6\") pod \"5c1c4cfe-7955-43ec-939d-450c1333a38c\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") "
Nov 03 15:39:17.218526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.218443    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume\") pod \"5c1c4cfe-7955-43ec-939d-450c1333a38c\" (UID: \"5c1c4cfe-7955-43ec-939d-450c1333a38c\") "
Nov 03 15:39:17.218747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.218720    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume" pod "5c1c4cfe-7955-43ec-939d-450c1333a38c" (UID: "5c1c4cfe-7955-43ec-939d-450c1333a38c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:39:17.219236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.219197    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c1c4cfe-7955-43ec-939d-450c1333a38c-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:17.220700 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.220678    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6" pod "5c1c4cfe-7955-43ec-939d-450c1333a38c" (UID: "5c1c4cfe-7955-43ec-939d-450c1333a38c"). InnerVolumeSpecName "kube-api-access-bllm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:39:17.320357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.320260    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume\") pod \"fb3e5f81-f567-4156-b6ac-981a1d962408\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") "
Nov 03 15:39:17.320528 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.320361    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m\" (UniqueName: \"kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m\") pod \"fb3e5f81-f567-4156-b6ac-981a1d962408\" (UID: \"fb3e5f81-f567-4156-b6ac-981a1d962408\") "
Nov 03 15:39:17.321053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.321026    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume" pod "fb3e5f81-f567-4156-b6ac-981a1d962408" (UID: "fb3e5f81-f567-4156-b6ac-981a1d962408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:39:17.321164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.321149    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e5f81-f567-4156-b6ac-981a1d962408-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:17.321220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.321178    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllm6\" (UniqueName: \"kubernetes.io/projected/5c1c4cfe-7955-43ec-939d-450c1333a38c-kube-api-access-bllm6\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:17.323344 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.323319    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m" pod "fb3e5f81-f567-4156-b6ac-981a1d962408" (UID: "fb3e5f81-f567-4156-b6ac-981a1d962408"). InnerVolumeSpecName "kube-api-access-8tb9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:39:17.422435 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.422399    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tb9m\" (UniqueName: \"kubernetes.io/projected/fb3e5f81-f567-4156-b6ac-981a1d962408-kube-api-access-8tb9m\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:17.805992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.805585    4955 generic.go:356] "Generic (PLEG): container finished" podID="fb3e5f81-f567-4156-b6ac-981a1d962408" containerID="44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23" exitCode=0
Nov 03 15:39:17.805992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.805747    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds" event={"ID":"fb3e5f81-f567-4156-b6ac-981a1d962408","Type":"ContainerDied","Data":"44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23"}
Nov 03 15:39:17.805992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.805822    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds" event={"ID":"fb3e5f81-f567-4156-b6ac-981a1d962408","Type":"ContainerDied","Data":"bdefc8dfa12b95222d92195ca413ec474d871331ee7aac1151411ca8692029b2"}
Nov 03 15:39:17.805992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.805863    4955 scope.go:122] "RemoveContainer" containerID="44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23"
Nov 03 15:39:17.806662 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.806081    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"
Nov 03 15:39:17.897030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.894573    4955 generic.go:356] "Generic (PLEG): container finished" podID="5c1c4cfe-7955-43ec-939d-450c1333a38c" containerID="20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43" exitCode=0
Nov 03 15:39:17.897030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.894669    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r" event={"ID":"5c1c4cfe-7955-43ec-939d-450c1333a38c","Type":"ContainerDied","Data":"20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43"}
Nov 03 15:39:17.897030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.894713    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r" event={"ID":"5c1c4cfe-7955-43ec-939d-450c1333a38c","Type":"ContainerDied","Data":"65fa8838efa1cccbaeb2b7bd318665b0131e49520a35e3d3246bf27e6873fce2"}
Nov 03 15:39:17.897030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.894790    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"
Nov 03 15:39:17.930994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.927226    4955 scope.go:122] "RemoveContainer" containerID="44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23"
Nov 03 15:39:17.930994 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:39:17.927654    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23\": container with ID starting with 44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23 not found: ID does not exist" containerID="44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23"
Nov 03 15:39:17.930994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.927691    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23"} err="failed to get container status \"44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23\": rpc error: code = NotFound desc = could not find container \"44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23\": container with ID starting with 44e9c8b069222c354dc07c474999e74625ffcdac80f39561a7ab8da737558a23 not found: ID does not exist"
Nov 03 15:39:17.930994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.927731    4955 scope.go:122] "RemoveContainer" containerID="20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43"
Nov 03 15:39:17.976348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.974390    4955 scope.go:122] "RemoveContainer" containerID="20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43"
Nov 03 15:39:17.976348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:39:17.974859    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43\": container with ID starting with 20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43 not found: ID does not exist" containerID="20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43"
Nov 03 15:39:17.976348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:17.974919    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43"} err="failed to get container status \"20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43\": rpc error: code = NotFound desc = could not find container \"20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43\": container with ID starting with 20fc89afb0a3d74aef087a695f6cdf2b95b34aced05a0f2f88a100eb68aece43 not found: ID does not exist"
Nov 03 15:39:18.016878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:18.016777    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"]
Nov 03 15:39:18.066645 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:18.066609    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-8a0e5a-6d9df7b647-kkrds"]
Nov 03 15:39:18.130668 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:18.130631    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"]
Nov 03 15:39:18.136099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:18.136071    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-8a0e5a-65bbf5dbc-m5g5r"]
Nov 03 15:39:19.207137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:19.207101    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1c4cfe-7955-43ec-939d-450c1333a38c" path="/var/lib/kubelet/pods/5c1c4cfe-7955-43ec-939d-450c1333a38c/volumes"
Nov 03 15:39:19.207552 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:19.207501    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3e5f81-f567-4156-b6ac-981a1d962408" path="/var/lib/kubelet/pods/fb3e5f81-f567-4156-b6ac-981a1d962408/volumes"
Nov 03 15:39:27.939774 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:39:27.939682    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:39:28.001881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:28.001850    4955 generic.go:356] "Generic (PLEG): container finished" podID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerID="12f052df44439d26c050b909271de51d590306cb2f1b9b2fee145379ae780440" exitCode=0
Nov 03 15:39:28.002062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:28.001925    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" event={"ID":"a019f010-05ff-47d9-8a42-bc507ab4780c","Type":"ContainerDied","Data":"12f052df44439d26c050b909271de51d590306cb2f1b9b2fee145379ae780440"}
Nov 03 15:39:28.002475 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:28.002458    4955 scope.go:122] "RemoveContainer" containerID="12f052df44439d26c050b909271de51d590306cb2f1b9b2fee145379ae780440"
Nov 03 15:39:28.004233 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:28.004200    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:39:28.004369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:28.004288    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:39:28.004369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:28.004336    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:39:29.017613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:29.017577    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" event={"ID":"a019f010-05ff-47d9-8a42-bc507ab4780c","Type":"ContainerStarted","Data":"6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a"}
Nov 03 15:39:29.985161 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:29.985121    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:39:29.985805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:29.985702    4955 patch_prober.go:29] interesting pod/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 15:39:29.986047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:29.986022    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 15:39:30.984699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:30.984666    4955 patch_prober.go:29] interesting pod/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 15:39:30.985163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:30.984732    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 15:39:31.984321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:31.984279    4955 patch_prober.go:29] interesting pod/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 15:39:31.984506 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:31.984351    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 15:39:32.984168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:32.984131    4955 patch_prober.go:29] interesting pod/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 15:39:32.984578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:32.984190    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 15:39:33.983952 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:33.983914    4955 patch_prober.go:29] interesting pod/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 15:39:33.984175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:33.984003    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 15:39:34.988718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:34.988680    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:39:34.989206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:34.989187    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:39:34.989805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:34.989788    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:39:46.414196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.414164    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"]
Nov 03 15:39:46.414611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.414386    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" podUID="336bce2a-2274-4867-b792-ea1e99b8e4a1" containerName="maas-api" containerID="cri-o://cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf" gracePeriod=30
Nov 03 15:39:46.764837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.764812    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:39:46.833213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.833174    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\") pod \"336bce2a-2274-4867-b792-ea1e99b8e4a1\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") "
Nov 03 15:39:46.833414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.833344    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca\") pod \"336bce2a-2274-4867-b792-ea1e99b8e4a1\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") "
Nov 03 15:39:46.833487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.833471    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs\") pod \"336bce2a-2274-4867-b792-ea1e99b8e4a1\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") "
Nov 03 15:39:46.833560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.833522    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52\" (UniqueName: \"kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52\") pod \"336bce2a-2274-4867-b792-ea1e99b8e4a1\" (UID: \"336bce2a-2274-4867-b792-ea1e99b8e4a1\") "
Nov 03 15:39:46.840656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.838399    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52" pod "336bce2a-2274-4867-b792-ea1e99b8e4a1" (UID: "336bce2a-2274-4867-b792-ea1e99b8e4a1"). InnerVolumeSpecName "kube-api-access-z4s52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:39:46.865279 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.857140    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca" pod "336bce2a-2274-4867-b792-ea1e99b8e4a1" (UID: "336bce2a-2274-4867-b792-ea1e99b8e4a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:39:46.948007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.940785    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336bce2a-2274-4867-b792-ea1e99b8e4a1-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:46.948007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.940824    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4s52\" (UniqueName: \"kubernetes.io/projected/336bce2a-2274-4867-b792-ea1e99b8e4a1-kube-api-access-z4s52\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:46.967209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.967169    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs" pod "336bce2a-2274-4867-b792-ea1e99b8e4a1" (UID: "336bce2a-2274-4867-b792-ea1e99b8e4a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:39:46.967397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:46.967244    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls" pod "336bce2a-2274-4867-b792-ea1e99b8e4a1" (UID: "336bce2a-2274-4867-b792-ea1e99b8e4a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:39:47.197345 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.172240    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:47.197345 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.172275    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336bce2a-2274-4867-b792-ea1e99b8e4a1-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:39:47.227846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.227809    4955 generic.go:356] "Generic (PLEG): container finished" podID="336bce2a-2274-4867-b792-ea1e99b8e4a1" containerID="cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf" exitCode=0
Nov 03 15:39:47.228053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.227867    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"
Nov 03 15:39:47.228053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.227906    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" event={"ID":"336bce2a-2274-4867-b792-ea1e99b8e4a1","Type":"ContainerDied","Data":"cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf"}
Nov 03 15:39:47.228053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.227999    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj" event={"ID":"336bce2a-2274-4867-b792-ea1e99b8e4a1","Type":"ContainerDied","Data":"9ee6ffade021ef2bb6ec36568e4f7e21227e1d055141232bcf460f97841c792b"}
Nov 03 15:39:47.228053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.228048    4955 scope.go:122] "RemoveContainer" containerID="cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf"
Nov 03 15:39:47.241392 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.241370    4955 scope.go:122] "RemoveContainer" containerID="cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf"
Nov 03 15:39:47.241707 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:39:47.241685    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf\": container with ID starting with cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf not found: ID does not exist" containerID="cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf"
Nov 03 15:39:47.241770 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.241716    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf"} err="failed to get container status \"cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf\": rpc error: code = NotFound desc = could not find container \"cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf\": container with ID starting with cabb8676c38a9d4627ebe6be7215a666dbc46e5cb2f60af5c9e2b15481ad38bf not found: ID does not exist"
Nov 03 15:39:47.442112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.442038    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"]
Nov 03 15:39:47.459748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:47.459666    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-8a0e5a-5c86884f47-mzhmj"]
Nov 03 15:39:49.206777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:49.206741    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336bce2a-2274-4867-b792-ea1e99b8e4a1" path="/var/lib/kubelet/pods/336bce2a-2274-4867-b792-ea1e99b8e4a1/volumes"
Nov 03 15:39:55.727585 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:55.727551    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"]
Nov 03 15:39:55.728246 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:55.727768    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" containerID="cri-o://1b73cd089ad8ea5a792d99cad2603b38f7c44a05dd9eeb7717072934fffb5712" gracePeriod=30
Nov 03 15:39:57.432062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432024    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"]
Nov 03 15:39:57.432701 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432686    4955 cpu_manager.go:414] "containerMap: removing container" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main"
Nov 03 15:39:57.432744 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432702    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main"
Nov 03 15:39:57.432744 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432732    4955 cpu_manager.go:414] "containerMap: removing container" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy"
Nov 03 15:39:57.432744 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432737    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432746    4955 cpu_manager.go:414] "containerMap: removing container" podUID="fb3e5f81-f567-4156-b6ac-981a1d962408" containerName="payload-pre-processing"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432751    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3e5f81-f567-4156-b6ac-981a1d962408" containerName="payload-pre-processing"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432758    4955 cpu_manager.go:414] "containerMap: removing container" podUID="336bce2a-2274-4867-b792-ea1e99b8e4a1" containerName="maas-api"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432762    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336bce2a-2274-4867-b792-ea1e99b8e4a1" containerName="maas-api"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432770    4955 cpu_manager.go:414] "containerMap: removing container" podUID="5c1c4cfe-7955-43ec-939d-450c1333a38c" containerName="payload-processing"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432775    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1c4cfe-7955-43ec-939d-450c1333a38c" containerName="payload-processing"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432785    4955 cpu_manager.go:414] "containerMap: removing container" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432790    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99202fd8-8caa-48a7-9a5f-df882c1098eb" containerName="istio-proxy"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432799    4955 cpu_manager.go:414] "containerMap: removing container" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy"
Nov 03 15:39:57.432843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432804    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy"
Nov 03 15:39:57.433170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432892    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb14684e-4893-48b6-9c94-ffa657288564" containerName="istio-proxy"
Nov 03 15:39:57.433170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432900    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3e5f81-f567-4156-b6ac-981a1d962408" containerName="payload-pre-processing"
Nov 03 15:39:57.433170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432908    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="336bce2a-2274-4867-b792-ea1e99b8e4a1" containerName="maas-api"
Nov 03 15:39:57.433170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432913    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1c4cfe-7955-43ec-939d-450c1333a38c" containerName="payload-processing"
Nov 03 15:39:57.433170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432922    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="76de34ed-49b4-405e-8161-978f3af6c0f2" containerName="istio-proxy"
Nov 03 15:39:57.433170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.432931    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="05297afe-2652-4157-8383-bcb8fc7c2851" containerName="main"
Nov 03 15:39:57.436343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.436325    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:57.507092 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.507058    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-78034f\""
Nov 03 15:39:57.576756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.576721    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-78034f-dockercfg-nmv8s\""
Nov 03 15:39:57.619095 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.597234    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:57.674214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.674173    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtps\" (UniqueName: \"kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps\") pod \"payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:57.682995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.624615    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"]
Nov 03 15:39:57.802582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.777051    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:57.806548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.802963    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtps\" (UniqueName: \"kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps\") pod \"payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:57.821174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.786305    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:57.934390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:57.934347    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtps\" (UniqueName: \"kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps\") pod \"payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:58.057484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:58.057451    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:58.099056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:58.098711    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:39:58.099056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:58.098765    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:39:58.122696 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:39:58.122366    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:39:58.546774 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:58.546741    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"]
Nov 03 15:39:58.547264 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:39:58.547241    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7a88c1_66e3_44a7_acf3_e1d3f2e315eb.slice/crio-d8fffc99ea339aefe65ded049d969c55f3d436a1b677ab9776d59e03d4ece8fb WatchSource:0}: Error finding container d8fffc99ea339aefe65ded049d969c55f3d436a1b677ab9776d59e03d4ece8fb: Status 404 returned error can't find the container with id d8fffc99ea339aefe65ded049d969c55f3d436a1b677ab9776d59e03d4ece8fb
Nov 03 15:39:59.361388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.361353    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk" event={"ID":"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb","Type":"ContainerStarted","Data":"70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008"}
Nov 03 15:39:59.361575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.361423    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:39:59.361575 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.361444    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk" event={"ID":"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb","Type":"ContainerStarted","Data":"d8fffc99ea339aefe65ded049d969c55f3d436a1b677ab9776d59e03d4ece8fb"}
Nov 03 15:39:59.407089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.407061    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"]
Nov 03 15:39:59.411399 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.411380    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.512995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.493091    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznrb\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.513385 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.513353    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.513539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.513522    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.522076 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.513685    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.522517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.522488    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.551001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.545176    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.551001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.545235    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.551001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.545286    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.551001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.545395    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.551001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.494394    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-90fcca-openshift-default-dockercfg-g7lcc\""
Nov 03 15:39:59.551001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.494096    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"]
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648116    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648178    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648228    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648329    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648443    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznrb\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648494    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648523    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648585    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.648619    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.649202    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.665413    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.667219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.666100    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.668034 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.667857    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.681063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.668451    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.681063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.672034    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.713543 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.713471    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.815256 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.815175    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznrb\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.818672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.818265    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-rt48g\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:39:59.895790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:39:59.880399    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk" podStartSLOduration=2.880369333 podStartE2EDuration="2.880369333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:39:59.540133253 +0000 UTC m=+1914.464951167" watchObservedRunningTime="2026-11-03 15:39:59.880369333 +0000 UTC m=+1914.805187320"
Nov 03 15:40:00.026124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:00.026043    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:40:00.505216 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:00.505181    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"]
Nov 03 15:40:00.561584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:00.561541    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:00.561930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:00.561652    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:00.561930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:00.561701    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:01.377782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.377741    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" event={"ID":"caab1123-8037-409b-af2e-31de277de6c8","Type":"ContainerStarted","Data":"592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"}
Nov 03 15:40:01.378022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.377813    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" event={"ID":"caab1123-8037-409b-af2e-31de277de6c8","Type":"ContainerStarted","Data":"e7902a72d426069cf949f962bb5556112c56bd160fc9c9b3e3cb59496ccbff6e"}
Nov 03 15:40:01.380345 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.380315    4955 generic.go:356] "Generic (PLEG): container finished" podID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerID="1b73cd089ad8ea5a792d99cad2603b38f7c44a05dd9eeb7717072934fffb5712" exitCode=0
Nov 03 15:40:01.380474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.380387    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" event={"ID":"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0","Type":"ContainerDied","Data":"1b73cd089ad8ea5a792d99cad2603b38f7c44a05dd9eeb7717072934fffb5712"}
Nov 03 15:40:01.380474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.380469    4955 scope.go:122] "RemoveContainer" containerID="1ec61e9595c9a67566273d6a68a33e402ae0fc669a4f738193ec2bd04a11ef71"
Nov 03 15:40:01.459332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.459305    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:40:01.517936 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.517851    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.517936 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.517919    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518008    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518046    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518143    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518213 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518162    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:01.518442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518234    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518263    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518362    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518400    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token\") pod \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\" (UID: \"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0\") "
Nov 03 15:40:01.518442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518429    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:01.518687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518457    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:01.518687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518556    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:01.518687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.518575    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:01.519206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.519180    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.519311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.519209    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.519311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.519224    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.519311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.519237    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.519311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.519251    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.520307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.520279    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:01.520470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.520452    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "kube-api-access-d7klj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:01.520583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.520567    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:01.520729 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.520716    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo" pod "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" (UID: "b6b13cbb-e8e7-4f76-a198-53105c6b4ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:40:01.563620 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.561597    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podStartSLOduration=2.561581527 podStartE2EDuration="2.561581527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:01.558535011 +0000 UTC m=+1916.483352922" watchObservedRunningTime="2026-11-03 15:40:01.561581527 +0000 UTC m=+1916.486399490"
Nov 03 15:40:01.620241 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.620210    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.620241 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.620238    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.620241 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.620247    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:01.620489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:01.620256    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7klj\" (UniqueName: \"kubernetes.io/projected/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0-kube-api-access-d7klj\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:02.026650 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.026609    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:40:02.054038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.046306    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:40:02.388275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.388195    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"
Nov 03 15:40:02.388275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.388236    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr" event={"ID":"b6b13cbb-e8e7-4f76-a198-53105c6b4ab0","Type":"ContainerDied","Data":"0ded5ca581b95ff9e4b11c5b01fbd1c18222a95a50a9f1a521d7bac5d4dc7996"}
Nov 03 15:40:02.388501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.388306    4955 scope.go:122] "RemoveContainer" containerID="1b73cd089ad8ea5a792d99cad2603b38f7c44a05dd9eeb7717072934fffb5712"
Nov 03 15:40:02.388707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.388664    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:40:02.389925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.389907    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:40:02.702679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.702647    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"]
Nov 03 15:40:02.705732 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:02.705706    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-8a0e5a-openshift-default-6574d747bf-v5cqr"]
Nov 03 15:40:03.207077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:03.207044    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" path="/var/lib/kubelet/pods/b6b13cbb-e8e7-4f76-a198-53105c6b4ab0/volumes"
Nov 03 15:40:08.670163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:08.670113    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:40:08.670578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:08.670190    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:40:08.921019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:08.920923    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"]
Nov 03 15:40:09.034640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.034541    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"]
Nov 03 15:40:09.533584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.533549    4955 generic.go:356] "Generic (PLEG): container finished" podID="b578c022-7306-43d9-8886-4f16c09db6c1" containerID="4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86" exitCode=0
Nov 03 15:40:09.533863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.533614    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" event={"ID":"b578c022-7306-43d9-8886-4f16c09db6c1","Type":"ContainerDied","Data":"4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"}
Nov 03 15:40:09.534368 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.534334    4955 scope.go:122] "RemoveContainer" containerID="4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"
Nov 03 15:40:09.536226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.536194    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:09.536354 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.536262    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:09.536354 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:09.536296    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:10.371788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:10.371758    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:40:10.883011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:10.868339    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" event={"ID":"b578c022-7306-43d9-8886-4f16c09db6c1","Type":"ContainerStarted","Data":"e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede"}
Nov 03 15:40:11.045907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:11.045474    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:40:11.045907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:11.045605    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 15:40:11.045907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:11.045650    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 15:40:12.054126 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:12.054088    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 15:40:12.054542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:12.054150    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 15:40:13.052995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:13.052325    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 15:40:13.052995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:13.052394    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 15:40:13.921791 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:13.921744    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl" podUID="99ad851e-cfa6-4b35-9a9a-b3947af80cad" containerName="payload-processing" containerID="cri-o://b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da" gracePeriod=25
Nov 03 15:40:14.054795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.054070    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss" podUID="d8ee3799-2775-47d9-8589-9ef649bbd2a2" containerName="payload-pre-processing" containerID="cri-o://f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024" gracePeriod=25
Nov 03 15:40:14.059011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.058980    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 15:40:14.059276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.059232    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 15:40:14.292431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.292399    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:40:14.388561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.388449    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf\" (UniqueName: \"kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf\") pod \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") "
Nov 03 15:40:14.388800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.388581    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume\") pod \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\" (UID: \"99ad851e-cfa6-4b35-9a9a-b3947af80cad\") "
Nov 03 15:40:14.389428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.389398    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume" pod "99ad851e-cfa6-4b35-9a9a-b3947af80cad" (UID: "99ad851e-cfa6-4b35-9a9a-b3947af80cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:14.391624 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.391596    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf" pod "99ad851e-cfa6-4b35-9a9a-b3947af80cad" (UID: "99ad851e-cfa6-4b35-9a9a-b3947af80cad"). InnerVolumeSpecName "kube-api-access-6d2cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:14.424841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.424809    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:40:14.490232 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.490131    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r\" (UniqueName: \"kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r\") pod \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") "
Nov 03 15:40:14.490418 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.490307    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume\") pod \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\" (UID: \"d8ee3799-2775-47d9-8589-9ef649bbd2a2\") "
Nov 03 15:40:14.490915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.490892    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad851e-cfa6-4b35-9a9a-b3947af80cad-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:14.491040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.490921    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2cf\" (UniqueName: \"kubernetes.io/projected/99ad851e-cfa6-4b35-9a9a-b3947af80cad-kube-api-access-6d2cf\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:14.491040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.491026    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume" pod "d8ee3799-2775-47d9-8589-9ef649bbd2a2" (UID: "d8ee3799-2775-47d9-8589-9ef649bbd2a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:14.492591 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.492568    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r" pod "d8ee3799-2775-47d9-8589-9ef649bbd2a2" (UID: "d8ee3799-2775-47d9-8589-9ef649bbd2a2"). InnerVolumeSpecName "kube-api-access-dkk8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:14.592195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.592160    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkk8r\" (UniqueName: \"kubernetes.io/projected/d8ee3799-2775-47d9-8589-9ef649bbd2a2-kube-api-access-dkk8r\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:14.592195 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.592190    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ee3799-2775-47d9-8589-9ef649bbd2a2-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:14.895036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.894999    4955 generic.go:356] "Generic (PLEG): container finished" podID="d8ee3799-2775-47d9-8589-9ef649bbd2a2" containerID="f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024" exitCode=0
Nov 03 15:40:14.895180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.895061    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"
Nov 03 15:40:14.895180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.895083    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss" event={"ID":"d8ee3799-2775-47d9-8589-9ef649bbd2a2","Type":"ContainerDied","Data":"f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024"}
Nov 03 15:40:14.895180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.895152    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss" event={"ID":"d8ee3799-2775-47d9-8589-9ef649bbd2a2","Type":"ContainerDied","Data":"fea6319d16df2b341fd29daba6ae5deef03f84014c9fc34e5fbd020cf6099e5e"}
Nov 03 15:40:14.895180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.895176    4955 scope.go:122] "RemoveContainer" containerID="f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024"
Nov 03 15:40:14.896635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.896565    4955 generic.go:356] "Generic (PLEG): container finished" podID="99ad851e-cfa6-4b35-9a9a-b3947af80cad" containerID="b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da" exitCode=0
Nov 03 15:40:14.896635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.896605    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"
Nov 03 15:40:14.896721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.896654    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl" event={"ID":"99ad851e-cfa6-4b35-9a9a-b3947af80cad","Type":"ContainerDied","Data":"b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da"}
Nov 03 15:40:14.896721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.896691    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl" event={"ID":"99ad851e-cfa6-4b35-9a9a-b3947af80cad","Type":"ContainerDied","Data":"4eb8a16b8033262a8d2f0af01fa2e7da0fd4b5410f22a3d292ff174e3c089bae"}
Nov 03 15:40:14.908035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.908019    4955 scope.go:122] "RemoveContainer" containerID="f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024"
Nov 03 15:40:14.908362 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:14.908338    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024\": container with ID starting with f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024 not found: ID does not exist" containerID="f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024"
Nov 03 15:40:14.908440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.908366    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024"} err="failed to get container status \"f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024\": rpc error: code = NotFound desc = could not find container \"f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024\": container with ID starting with f218873f70d6ed3c381806f7242f722aad71caee866d2e22ae59c78cfba98024 not found: ID does not exist"
Nov 03 15:40:14.908440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.908383    4955 scope.go:122] "RemoveContainer" containerID="b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da"
Nov 03 15:40:14.923317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.923089    4955 scope.go:122] "RemoveContainer" containerID="b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da"
Nov 03 15:40:14.923602 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:14.923366    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da\": container with ID starting with b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da not found: ID does not exist" containerID="b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da"
Nov 03 15:40:14.923602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:14.923390    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da"} err="failed to get container status \"b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da\": rpc error: code = NotFound desc = could not find container \"b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da\": container with ID starting with b1e31a3c6440e2bf5c04f552cffeb6fdedd2df17c0d01523f5a66e91cc94f0da not found: ID does not exist"
Nov 03 15:40:15.042716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.042371    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 15:40:15.043067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.043024    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 15:40:15.084675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.084640    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"]
Nov 03 15:40:15.094390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.094361    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-7b3fd3-d66c6778c-zq7pl"]
Nov 03 15:40:15.207870 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.207776    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ad851e-cfa6-4b35-9a9a-b3947af80cad" path="/var/lib/kubelet/pods/99ad851e-cfa6-4b35-9a9a-b3947af80cad/volumes"
Nov 03 15:40:15.208369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.208346    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"]
Nov 03 15:40:15.218734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:15.218711    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-7b3fd3-587c8f5f95-5dkss"]
Nov 03 15:40:16.041697 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:16.041655    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 15:40:16.042134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:16.041727    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 15:40:17.046524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:17.046486    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:40:17.047369 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:17.047343    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:40:17.047694 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:17.047674    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:40:17.206191 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:17.206162    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ee3799-2775-47d9-8589-9ef649bbd2a2" path="/var/lib/kubelet/pods/d8ee3799-2775-47d9-8589-9ef649bbd2a2/volumes"
Nov 03 15:40:19.404327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:19.404293    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:40:26.599150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:26.599107    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"]
Nov 03 15:40:27.984683 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:27.984640    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:40:29.140873 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.140842    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"]
Nov 03 15:40:29.141330 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.141075    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" podUID="7d06218e-f342-4a0c-8968-f59b54683360" containerName="maas-api" containerID="cri-o://eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547" gracePeriod=30
Nov 03 15:40:29.480136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.480108    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:40:29.573718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.573676    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc\" (UniqueName: \"kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc\") pod \"7d06218e-f342-4a0c-8968-f59b54683360\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") "
Nov 03 15:40:29.574124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.574105    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs\") pod \"7d06218e-f342-4a0c-8968-f59b54683360\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") "
Nov 03 15:40:29.574338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.574324    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\") pod \"7d06218e-f342-4a0c-8968-f59b54683360\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") "
Nov 03 15:40:29.574550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.574537    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca\") pod \"7d06218e-f342-4a0c-8968-f59b54683360\" (UID: \"7d06218e-f342-4a0c-8968-f59b54683360\") "
Nov 03 15:40:29.575814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.575788    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca" pod "7d06218e-f342-4a0c-8968-f59b54683360" (UID: "7d06218e-f342-4a0c-8968-f59b54683360"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:29.592746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.592687    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls" pod "7d06218e-f342-4a0c-8968-f59b54683360" (UID: "7d06218e-f342-4a0c-8968-f59b54683360"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:40:29.598498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.598142    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc" pod "7d06218e-f342-4a0c-8968-f59b54683360" (UID: "7d06218e-f342-4a0c-8968-f59b54683360"). InnerVolumeSpecName "kube-api-access-djhcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:29.598958 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.598920    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs" pod "7d06218e-f342-4a0c-8968-f59b54683360" (UID: "7d06218e-f342-4a0c-8968-f59b54683360"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:40:29.705037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.704928    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:29.705037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.705011    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d06218e-f342-4a0c-8968-f59b54683360-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:29.705037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.705030    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d06218e-f342-4a0c-8968-f59b54683360-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:29.705037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:29.705044    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhcc\" (UniqueName: \"kubernetes.io/projected/7d06218e-f342-4a0c-8968-f59b54683360-kube-api-access-djhcc\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:30.117511 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.117414    4955 generic.go:356] "Generic (PLEG): container finished" podID="7d06218e-f342-4a0c-8968-f59b54683360" containerID="eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547" exitCode=0
Nov 03 15:40:30.117900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.117873    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" event={"ID":"7d06218e-f342-4a0c-8968-f59b54683360","Type":"ContainerDied","Data":"eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547"}
Nov 03 15:40:30.118098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.118082    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772" event={"ID":"7d06218e-f342-4a0c-8968-f59b54683360","Type":"ContainerDied","Data":"0f733ce824ad0842a313c9f801f08d9a37342688eef9667a2f323adca45b5622"}
Nov 03 15:40:30.118229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.118208    4955 scope.go:122] "RemoveContainer" containerID="eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547"
Nov 03 15:40:30.118489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.118472    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"
Nov 03 15:40:30.174470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.172208    4955 scope.go:122] "RemoveContainer" containerID="eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547"
Nov 03 15:40:30.177251 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:30.177219    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547\": container with ID starting with eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547 not found: ID does not exist" containerID="eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547"
Nov 03 15:40:30.177439 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.177412    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547"} err="failed to get container status \"eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547\": rpc error: code = NotFound desc = could not find container \"eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547\": container with ID starting with eb771230a03c98b0322e362c0145d95dd24be56e381568d6b7a2e72ce9b61547 not found: ID does not exist"
Nov 03 15:40:30.342131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.342101    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"]
Nov 03 15:40:30.392652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:30.392622    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f78fd6-b89c84bbd-rs772"]
Nov 03 15:40:31.211805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:31.211770    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d06218e-f342-4a0c-8968-f59b54683360" path="/var/lib/kubelet/pods/7d06218e-f342-4a0c-8968-f59b54683360/volumes"
Nov 03 15:40:31.599877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:31.599765    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" podUID="bd280704-5597-4533-8446-3b9a126f9483" containerName="payload-pre-processing" containerID="cri-o://d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b" gracePeriod=25
Nov 03 15:40:32.484320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:32.484282    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" podUID="bd280704-5597-4533-8446-3b9a126f9483" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:40:38.803764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:38.803729    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"]
Nov 03 15:40:38.804261 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:38.803937    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" containerID="cri-o://53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150" gracePeriod=30
Nov 03 15:40:38.820038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:38.820004    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"]
Nov 03 15:40:38.820268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:38.820232    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy" containerID="cri-o://738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658" gracePeriod=30
Nov 03 15:40:40.060104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:40.060067    4955 patch_prober.go:29] interesting pod/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:40:40.060513 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:40.060120    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:40:42.483916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:42.483880    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" podUID="bd280704-5597-4533-8446-3b9a126f9483" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:40:44.100395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.100365    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:40:44.141591 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.141563    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:40:44.250453 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.250414    4955 generic.go:356] "Generic (PLEG): container finished" podID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerID="53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150" exitCode=0
Nov 03 15:40:44.250658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.250465    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"
Nov 03 15:40:44.250658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.250500    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" event={"ID":"7091eb22-7fc0-4a2b-95a8-3a7199144cd1","Type":"ContainerDied","Data":"53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150"}
Nov 03 15:40:44.250658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.250568    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g" event={"ID":"7091eb22-7fc0-4a2b-95a8-3a7199144cd1","Type":"ContainerDied","Data":"1e15e85fc0f8baeaf78bf18247df5bb2048affd839a64d86263a53962b12267c"}
Nov 03 15:40:44.250658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.250600    4955 scope.go:122] "RemoveContainer" containerID="53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150"
Nov 03 15:40:44.252478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.252442    4955 generic.go:356] "Generic (PLEG): container finished" podID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerID="738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658" exitCode=0
Nov 03 15:40:44.252614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.252486    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"
Nov 03 15:40:44.252614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.252555    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" event={"ID":"5b84ae9d-e1cf-4abe-a2af-83108011e47f","Type":"ContainerDied","Data":"738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658"}
Nov 03 15:40:44.252614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.252592    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7" event={"ID":"5b84ae9d-e1cf-4abe-a2af-83108011e47f","Type":"ContainerDied","Data":"07922cfe1b68930eec1f9bcbb0d659aa1818326dace2968346c5ea13755cc92f"}
Nov 03 15:40:44.261055 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.261035    4955 scope.go:122] "RemoveContainer" containerID="53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150"
Nov 03 15:40:44.261361 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:44.261338    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150\": container with ID starting with 53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150 not found: ID does not exist" containerID="53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150"
Nov 03 15:40:44.261437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.261368    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150"} err="failed to get container status \"53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150\": rpc error: code = NotFound desc = could not find container \"53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150\": container with ID starting with 53d86a2ac05114a5b3f29cd3f4a2d41011d28a5a11ea4ff2220c85deceaab150 not found: ID does not exist"
Nov 03 15:40:44.261437 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.261387    4955 scope.go:122] "RemoveContainer" containerID="738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658"
Nov 03 15:40:44.270946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.270925    4955 scope.go:122] "RemoveContainer" containerID="738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658"
Nov 03 15:40:44.271224 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:44.271205    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658\": container with ID starting with 738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658 not found: ID does not exist" containerID="738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658"
Nov 03 15:40:44.271282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.271231    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658"} err="failed to get container status \"738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658\": rpc error: code = NotFound desc = could not find container \"738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658\": container with ID starting with 738a3565497d5307e265e4d4c76e251d78e6e54d05177dbbcd2870808f635658 not found: ID does not exist"
Nov 03 15:40:44.276726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276699    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.276838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276752    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.276838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276777    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.276838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276823    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276852    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276893    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276925    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.276980    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277056    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277075    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:44.277225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277099    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277144    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277198    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277272    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277099    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.277400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277279    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.277400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277351    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277400 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277389    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.277642 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277429    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277642 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277491    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data\") pod \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\" (UID: \"7091eb22-7fc0-4a2b-95a8-3a7199144cd1\") "
Nov 03 15:40:44.277642 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.277526    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data\") pod \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\" (UID: \"5b84ae9d-e1cf-4abe-a2af-83108011e47f\") "
Nov 03 15:40:44.278100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.278077    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.278218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.278109    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.278494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.278420    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.278494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.278440    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.278634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.278589    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.278902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.278873    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.279320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.279283    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:44.280118 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280095    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280136    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280153    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280168    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280182    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280195    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280208    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280222    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280234    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.280276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.280248    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.282583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282496    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.282583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282511    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:40:44.282583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282525    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:40:44.282583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282530    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "kube-api-access-vx4mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:44.282583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282560    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:44.282865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282598    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx" pod "5b84ae9d-e1cf-4abe-a2af-83108011e47f" (UID: "5b84ae9d-e1cf-4abe-a2af-83108011e47f"). InnerVolumeSpecName "kube-api-access-cpbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:44.282865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282750    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:40:44.282865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.282764    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token" pod "7091eb22-7fc0-4a2b-95a8-3a7199144cd1" (UID: "7091eb22-7fc0-4a2b-95a8-3a7199144cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381332    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4mk\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-kube-api-access-vx4mk\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381362    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbkx\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-kube-api-access-cpbkx\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381371    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381380    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381388    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381398    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7091eb22-7fc0-4a2b-95a8-3a7199144cd1-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381407    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.381411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.381415    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b84ae9d-e1cf-4abe-a2af-83108011e47f-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:44.678741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.678706    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"]
Nov 03 15:40:44.685618 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.685588    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-8wk5g"]
Nov 03 15:40:44.799170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.799132    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"]
Nov 03 15:40:44.804464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:44.804436    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f78fd6-openshift-default-74b46c4887-q49s7"]
Nov 03 15:40:45.205736 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:45.205699    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" path="/var/lib/kubelet/pods/5b84ae9d-e1cf-4abe-a2af-83108011e47f/volumes"
Nov 03 15:40:45.206460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:45.206439    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" path="/var/lib/kubelet/pods/7091eb22-7fc0-4a2b-95a8-3a7199144cd1/volumes"
Nov 03 15:40:48.560561 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.560527    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"]
Nov 03 15:40:48.561241 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561225    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561245    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561265    4955 cpu_manager.go:414] "containerMap: removing container" podUID="99ad851e-cfa6-4b35-9a9a-b3947af80cad" containerName="payload-processing"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561272    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ad851e-cfa6-4b35-9a9a-b3947af80cad" containerName="payload-processing"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561279    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7d06218e-f342-4a0c-8968-f59b54683360" containerName="maas-api"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561284    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d06218e-f342-4a0c-8968-f59b54683360" containerName="maas-api"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561304    4955 cpu_manager.go:414] "containerMap: removing container" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy"
Nov 03 15:40:48.561309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561308    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561317    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561324    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561337    4955 cpu_manager.go:414] "containerMap: removing container" podUID="d8ee3799-2775-47d9-8589-9ef649bbd2a2" containerName="payload-pre-processing"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561342    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ee3799-2775-47d9-8589-9ef649bbd2a2" containerName="payload-pre-processing"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561357    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561361    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561477    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561496    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7091eb22-7fc0-4a2b-95a8-3a7199144cd1" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561504    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ad851e-cfa6-4b35-9a9a-b3947af80cad" containerName="payload-processing"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561510    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d06218e-f342-4a0c-8968-f59b54683360" containerName="maas-api"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561518    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b84ae9d-e1cf-4abe-a2af-83108011e47f" containerName="istio-proxy"
Nov 03 15:40:48.561524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.561525    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ee3799-2775-47d9-8589-9ef649bbd2a2" containerName="payload-pre-processing"
Nov 03 15:40:48.566541 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.566523    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.659028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.642467    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"]
Nov 03 15:40:48.670254 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.670216    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-75edeb-openshift-default-dockercfg-x5zgk\""
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.681912    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682010    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdvl\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682074    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682198    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682233    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682268    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682302    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682365    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.683992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.682442    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.808006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.800885    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.808348    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815255    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815325    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdvl\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815380    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815515    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815541    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815576    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.815612    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.816175    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.816605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.816473    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.825061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.825014    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.826176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.826109    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.826337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.826193    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:48.829545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:48.829511    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:49.039000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.031356    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:49.048718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.048688    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdvl\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:49.090043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.087642    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token\") pod \"e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") " pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:49.200516 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.200481    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:49.666170 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:40:49.666137    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf909bbbf_ca76_48ac_9e32_4aa4d5613c67.slice/crio-e44624cc64febcd05cae71f58ce8f3b866f3446ec3633eb15d4ca6406e7829b3 WatchSource:0}: Error finding container e44624cc64febcd05cae71f58ce8f3b866f3446ec3633eb15d4ca6406e7829b3: Status 404 returned error can't find the container with id e44624cc64febcd05cae71f58ce8f3b866f3446ec3633eb15d4ca6406e7829b3
Nov 03 15:40:49.666170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.666158    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"]
Nov 03 15:40:49.668698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.668665    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:49.668781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.668734    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:49.668781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:49.668762    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:50.317722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:50.317685    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" event={"ID":"f909bbbf-ca76-48ac-9e32-4aa4d5613c67","Type":"ContainerStarted","Data":"c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"}
Nov 03 15:40:50.317900 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:50.317739    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" event={"ID":"f909bbbf-ca76-48ac-9e32-4aa4d5613c67","Type":"ContainerStarted","Data":"e44624cc64febcd05cae71f58ce8f3b866f3446ec3633eb15d4ca6406e7829b3"}
Nov 03 15:40:51.208568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:51.208539    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:51.208944 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:51.208630    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:51.325335 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:51.325306    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:51.326537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:51.326518    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:40:51.411826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:51.411769    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podStartSLOduration=3.411755286 podStartE2EDuration="3.411755286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:50.454442967 +0000 UTC m=+1965.379260920" watchObservedRunningTime="2026-11-03 15:40:51.411755286 +0000 UTC m=+1966.336573199"
Nov 03 15:40:51.949112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:51.949085    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:40:52.005220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.005190    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn\" (UniqueName: \"kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn\") pod \"bd280704-5597-4533-8446-3b9a126f9483\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") "
Nov 03 15:40:52.005440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.005248    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume\") pod \"bd280704-5597-4533-8446-3b9a126f9483\" (UID: \"bd280704-5597-4533-8446-3b9a126f9483\") "
Nov 03 15:40:52.005714 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.005690    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume" pod "bd280704-5597-4533-8446-3b9a126f9483" (UID: "bd280704-5597-4533-8446-3b9a126f9483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:40:52.005824 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.005808    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd280704-5597-4533-8446-3b9a126f9483-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:52.007583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.007554    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn" pod "bd280704-5597-4533-8446-3b9a126f9483" (UID: "bd280704-5597-4533-8446-3b9a126f9483"). InnerVolumeSpecName "kube-api-access-78ssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:40:52.108911 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.108874    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ssn\" (UniqueName: \"kubernetes.io/projected/bd280704-5597-4533-8446-3b9a126f9483-kube-api-access-78ssn\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:40:52.334313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.334217    4955 generic.go:356] "Generic (PLEG): container finished" podID="bd280704-5597-4533-8446-3b9a126f9483" containerID="d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b" exitCode=0
Nov 03 15:40:52.334313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.334274    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"
Nov 03 15:40:52.334313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.334298    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" event={"ID":"bd280704-5597-4533-8446-3b9a126f9483","Type":"ContainerDied","Data":"d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b"}
Nov 03 15:40:52.334840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.334350    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf" event={"ID":"bd280704-5597-4533-8446-3b9a126f9483","Type":"ContainerDied","Data":"c3754475131b4710d65c56d1a56b9cfaf8e4e500ad8e412ce9683f55e27837dc"}
Nov 03 15:40:52.334840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.334376    4955 scope.go:122] "RemoveContainer" containerID="d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b"
Nov 03 15:40:52.345317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.345295    4955 scope.go:122] "RemoveContainer" containerID="d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b"
Nov 03 15:40:52.345641 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:52.345620    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b\": container with ID starting with d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b not found: ID does not exist" containerID="d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b"
Nov 03 15:40:52.345687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.345651    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b"} err="failed to get container status \"d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b\": rpc error: code = NotFound desc = could not find container \"d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b\": container with ID starting with d15582bb4cec9296bf3c333e76ebf006d66409043659c0d50d852837d2e7c97b not found: ID does not exist"
Nov 03 15:40:52.513474 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.513440    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"]
Nov 03 15:40:52.518757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:52.518723    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5cad072f-fbb45455f-7dvjf"]
Nov 03 15:40:53.209378 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:53.209335    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd280704-5597-4533-8446-3b9a126f9483" path="/var/lib/kubelet/pods/bd280704-5597-4533-8446-3b9a126f9483/volumes"
Nov 03 15:40:56.364946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.364914    4955 generic.go:356] "Generic (PLEG): container finished" podID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerID="66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57" exitCode=0
Nov 03 15:40:56.365463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.365023    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" event={"ID":"2ab2ea66-3da5-4b85-979f-a40a2253b523","Type":"ContainerDied","Data":"66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"}
Nov 03 15:40:56.365530 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.365512    4955 scope.go:122] "RemoveContainer" containerID="66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"
Nov 03 15:40:56.367491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.367460    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:56.367608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.367532    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:56.367608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.367566    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:40:56.679935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:56.679903    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:40:57.047380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.046587    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"]
Nov 03 15:40:57.047597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.047545    4955 cpu_manager.go:414] "containerMap: removing container" podUID="bd280704-5597-4533-8446-3b9a126f9483" containerName="payload-pre-processing"
Nov 03 15:40:57.047672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.047602    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd280704-5597-4533-8446-3b9a126f9483" containerName="payload-pre-processing"
Nov 03 15:40:57.047836 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.047814    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b13cbb-e8e7-4f76-a198-53105c6b4ab0" containerName="istio-proxy"
Nov 03 15:40:57.047930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.047852    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd280704-5597-4533-8446-3b9a126f9483" containerName="payload-pre-processing"
Nov 03 15:40:57.051788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.051761    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.112433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.111884    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-75edeb\""
Nov 03 15:40:57.112433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.111945    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-75edeb-dockercfg-t6q2m\""
Nov 03 15:40:57.114892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.114857    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"]
Nov 03 15:40:57.357073 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.320018    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7wd\" (UniqueName: \"kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd\") pod \"payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.357073 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.320212    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.421793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.421726    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.422419 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.421947    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7wd\" (UniqueName: \"kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd\") pod \"payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.423000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.422959    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.466484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.466445    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" event={"ID":"2ab2ea66-3da5-4b85-979f-a40a2253b523","Type":"ContainerStarted","Data":"e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9"}
Nov 03 15:40:57.467622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.467583    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"]
Nov 03 15:40:57.473932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.473908    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"]
Nov 03 15:40:57.474230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.474051    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.485290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.485205    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7wd\" (UniqueName: \"kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd\") pod \"payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.624788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.624681    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.625012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.624935    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br44h\" (UniqueName: \"kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h\") pod \"payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.649996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.649197    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:40:57.649996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.649348    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 15:40:57.649996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.649409    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 15:40:57.667170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.667121    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:57.726198 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.726154    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.726382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.726361    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br44h\" (UniqueName: \"kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h\") pod \"payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.727146 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.727093    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.790005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.789947    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br44h\" (UniqueName: \"kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h\") pod \"payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.802922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:57.802694    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:57.971496 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:40:57.971449    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:40:58.059628 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:40:58.059597    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b4f369_9910_4569_81f3_3cd7866ec52d.slice/crio-95c3ee073655d74560594ee904fa626ec0867170289899cb5cb3b4741fcb58d2 WatchSource:0}: Error finding container 95c3ee073655d74560594ee904fa626ec0867170289899cb5cb3b4741fcb58d2: Status 404 returned error can't find the container with id 95c3ee073655d74560594ee904fa626ec0867170289899cb5cb3b4741fcb58d2
Nov 03 15:40:58.060581 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.060506    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"]
Nov 03 15:40:58.219576 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.218310    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"]
Nov 03 15:40:58.222674 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:40:58.222641    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7062e4ce_9e99_4133_b1c7_3abee02a3e53.slice/crio-cd4515a8730405ea9ff71689d4825d899b5d035228df6bb5e043e6d988fb41d5 WatchSource:0}: Error finding container cd4515a8730405ea9ff71689d4825d899b5d035228df6bb5e043e6d988fb41d5: Status 404 returned error can't find the container with id cd4515a8730405ea9ff71689d4825d899b5d035228df6bb5e043e6d988fb41d5
Nov 03 15:40:58.397271 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.397234    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6" event={"ID":"01b4f369-9910-4569-81f3-3cd7866ec52d","Type":"ContainerStarted","Data":"c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8"}
Nov 03 15:40:58.397463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.397298    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6" event={"ID":"01b4f369-9910-4569-81f3-3cd7866ec52d","Type":"ContainerStarted","Data":"95c3ee073655d74560594ee904fa626ec0867170289899cb5cb3b4741fcb58d2"}
Nov 03 15:40:58.397463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.397327    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:40:58.399017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.398985    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5" event={"ID":"7062e4ce-9e99-4133-b1c7-3abee02a3e53","Type":"ContainerStarted","Data":"46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4"}
Nov 03 15:40:58.399192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.399037    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5" event={"ID":"7062e4ce-9e99-4133-b1c7-3abee02a3e53","Type":"ContainerStarted","Data":"cd4515a8730405ea9ff71689d4825d899b5d035228df6bb5e043e6d988fb41d5"}
Nov 03 15:40:58.399192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.399180    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:40:58.522728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.522632    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6" podStartSLOduration=2.522617575 podStartE2EDuration="2.522617575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:58.518436403 +0000 UTC m=+1973.443254313" watchObservedRunningTime="2026-11-03 15:40:58.522617575 +0000 UTC m=+1973.447435540"
Nov 03 15:40:58.660997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.654375    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 15:40:58.660997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.654430    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 15:40:58.667010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:58.666079    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5" podStartSLOduration=1.666053816 podStartE2EDuration="1.666053816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:40:58.654084093 +0000 UTC m=+1973.578902001" watchObservedRunningTime="2026-11-03 15:40:58.666053816 +0000 UTC m=+1973.590871730"
Nov 03 15:40:59.654115 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:59.654076    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 15:40:59.654721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:40:59.654690    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 15:41:00.654394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:00.654352    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 15:41:00.654828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:00.654407    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 15:41:01.655165 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:01.655123    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 15:41:01.655680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:01.655187    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 15:41:02.648688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:02.648648    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 15:41:02.648949 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:02.648918    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 15:41:03.641074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:03.641039    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:41:03.641457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:03.641303    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:41:03.642157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:03.642139    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:41:05.893512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:05.893477    4955 scope.go:122] "RemoveContainer" containerID="4384e74ed0719693204a0e2eb73bc299e8c7a91f25bfa431fa6b00b6f7de040a"
Nov 03 15:41:07.706139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.706109    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"]
Nov 03 15:41:07.712039 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.712021    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.795368 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.795331    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.795583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.795561    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xb97\" (UniqueName: \"kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.795669 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.795655    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.795725 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.795698    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.815493 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.815461    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-78034f-serving-cert\""
Nov 03 15:41:07.818711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.818685    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"]
Nov 03 15:41:07.897375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.897337    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xb97\" (UniqueName: \"kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.897586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.897418    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.897586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.897476    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.897586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.897553    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.897586 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:07.897564    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-serving-cert: secret "maas-api-e2e-shared-b-w0-78034f-serving-cert" not found
Nov 03 15:41:07.897788 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:07.897621    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls podName:bd18e2cb-ed33-4461-b18f-1458b96322ed nodeName:}" failed. No retries permitted until 2026-11-03 15:41:08.397606295 +0000 UTC m=+1983.322424222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls") pod "maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" (UID: "bd18e2cb-ed33-4461-b18f-1458b96322ed") : secret "maas-api-e2e-shared-b-w0-78034f-serving-cert" not found
Nov 03 15:41:07.898257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.898238    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:07.900249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:07.900232    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:08.016268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:08.016172    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xb97\" (UniqueName: \"kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:08.422797 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:08.422761    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:08.425322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:08.425300    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:08.625549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:08.625511    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:09.006351 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:09.006317    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"]
Nov 03 15:41:09.007411 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:41:09.007386    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd18e2cb_ed33_4461_b18f_1458b96322ed.slice/crio-c403f3affddbad93d5dce8b872b586285640a6066f16e8b4705755c3ee4ca25b WatchSource:0}: Error finding container c403f3affddbad93d5dce8b872b586285640a6066f16e8b4705755c3ee4ca25b: Status 404 returned error can't find the container with id c403f3affddbad93d5dce8b872b586285640a6066f16e8b4705755c3ee4ca25b
Nov 03 15:41:09.009442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:09.009424    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:41:09.408746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:09.408715    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:41:09.408914 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:09.408783    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:41:09.470868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:09.470835    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" event={"ID":"bd18e2cb-ed33-4461-b18f-1458b96322ed","Type":"ContainerStarted","Data":"c403f3affddbad93d5dce8b872b586285640a6066f16e8b4705755c3ee4ca25b"}
Nov 03 15:41:10.481406 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:10.481368    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" event={"ID":"bd18e2cb-ed33-4461-b18f-1458b96322ed","Type":"ContainerStarted","Data":"bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f"}
Nov 03 15:41:10.481894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:10.481462    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:10.600559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:10.600497    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" podStartSLOduration=2.635747489 podStartE2EDuration="3.600478948s" totalImagesPullingTime="964.731459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:41:10.600314828 +0000 UTC m=+1985.525132743" watchObservedRunningTime="2026-11-03 15:41:10.600478948 +0000 UTC m=+1985.525296864"
Nov 03 15:41:16.495105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:16.495074    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:41:17.606790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.606752    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:41:17.700392 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.700357    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"]
Nov 03 15:41:17.704875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.704855    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.760485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.760449    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-90fcca-serving-cert\""
Nov 03 15:41:17.764644 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.764612    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"]
Nov 03 15:41:17.790860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.790825    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7lv\" (UniqueName: \"kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.791137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.791113    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.791189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.791151    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.791189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.791180    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.892918 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.892883    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.893114 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.892958    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7lv\" (UniqueName: \"kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.893194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.893181    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.893239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.893207    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.893308 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:17.893290    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-serving-cert: secret "maas-api-e2e-api-b-90fcca-serving-cert" not found
Nov 03 15:41:17.893358 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:17.893348    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls podName:4a6a00f6-f94e-4663-8398-cb49f8d080c6 nodeName:}" failed. No retries permitted until 2026-11-03 15:41:18.393333632 +0000 UTC m=+1993.318151523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls") pod "maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" (UID: "4a6a00f6-f94e-4663-8398-cb49f8d080c6") : secret "maas-api-e2e-api-b-90fcca-serving-cert" not found
Nov 03 15:41:17.893780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.893764    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:17.895674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:17.895648    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:18.010844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:18.010787    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7lv\" (UniqueName: \"kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:18.406260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:18.406219    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:18.408784 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:18.408763    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\") pod \"maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:18.622181 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:18.622150    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:19.004667 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:41:19.004639    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6a00f6_f94e_4663_8398_cb49f8d080c6.slice/crio-19dc483f8a7382040209eaa3fa42ebfacd68a4deda1c63ecdbee6839d478de67 WatchSource:0}: Error finding container 19dc483f8a7382040209eaa3fa42ebfacd68a4deda1c63ecdbee6839d478de67: Status 404 returned error can't find the container with id 19dc483f8a7382040209eaa3fa42ebfacd68a4deda1c63ecdbee6839d478de67
Nov 03 15:41:19.004830 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:19.004666    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"]
Nov 03 15:41:19.553053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:19.553013    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" event={"ID":"4a6a00f6-f94e-4663-8398-cb49f8d080c6","Type":"ContainerStarted","Data":"19dc483f8a7382040209eaa3fa42ebfacd68a4deda1c63ecdbee6839d478de67"}
Nov 03 15:41:20.656282 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:20.656208    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" event={"ID":"4a6a00f6-f94e-4663-8398-cb49f8d080c6","Type":"ContainerStarted","Data":"2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea"}
Nov 03 15:41:20.658337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:20.658305    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:21.333996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:21.244584    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:21.621214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:21.596069    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:21.828999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:21.749388    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Nov 03 15:41:21.830331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:21.774070    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:41:21.999478 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:21.999075    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:22.143141 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:22.143086    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 03 15:41:22.217821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:22.217775    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 15:41:22.245993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:22.243243    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:27.872427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:27.872394    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:41:27.943353 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:27.943316    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:41:27.992825 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:27.992769    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" podStartSLOduration=9.99257921 podStartE2EDuration="10.992754942s" totalImagesPullingTime="1.000175732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:41:20.815491733 +0000 UTC m=+1995.740309663" watchObservedRunningTime="2026-11-03 15:41:27.992754942 +0000 UTC m=+2002.917572855"
Nov 03 15:41:32.193692 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:32.193658    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"]
Nov 03 15:41:32.194150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:32.193882    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy" containerID="cri-o://0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf" gracePeriod=30
Nov 03 15:41:37.487807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.487779    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:41:37.595307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595260    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595507 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595475    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595519    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595570    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595614    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595647    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595737    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.595852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595817    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:41:37.595904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595843    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.596013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595836    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:41:37.596013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595849    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:41:37.596013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.595989    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs\") pod \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\" (UID: \"691ab01b-5d2e-4dd9-868e-c49f850e2e67\") "
Nov 03 15:41:37.596209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596189    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:41:37.596294 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596272    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:41:37.596888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596868    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.596950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596897    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.596950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596915    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.596950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596931    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.596950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.596946    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.597777 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.597754    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:41:37.598045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.598021    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:41:37.598545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.598525    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:41:37.598611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.598543    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv" pod "691ab01b-5d2e-4dd9-868e-c49f850e2e67" (UID: "691ab01b-5d2e-4dd9-868e-c49f850e2e67"). InnerVolumeSpecName "kube-api-access-fpxwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:41:37.698537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.698505    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.698537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.698537    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.698793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.698547    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxwv\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-kube-api-access-fpxwv\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.698793 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.698558    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691ab01b-5d2e-4dd9-868e-c49f850e2e67-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:41:37.966089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.966052    4955 generic.go:356] "Generic (PLEG): container finished" podID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerID="0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf" exitCode=0
Nov 03 15:41:37.966298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.966110    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"
Nov 03 15:41:37.966298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.966133    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" event={"ID":"691ab01b-5d2e-4dd9-868e-c49f850e2e67","Type":"ContainerDied","Data":"0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf"}
Nov 03 15:41:37.966298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.966202    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz" event={"ID":"691ab01b-5d2e-4dd9-868e-c49f850e2e67","Type":"ContainerDied","Data":"25e1f6aa9ee4bddf39fb93d86b5e589e29d2506b6aad0b523100b5c951615527"}
Nov 03 15:41:37.966298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.966245    4955 scope.go:122] "RemoveContainer" containerID="0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf"
Nov 03 15:41:37.977169 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.977144    4955 scope.go:122] "RemoveContainer" containerID="c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"
Nov 03 15:41:37.986471 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.986452    4955 scope.go:122] "RemoveContainer" containerID="0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf"
Nov 03 15:41:37.986785 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:37.986764    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf\": container with ID starting with 0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf not found: ID does not exist" containerID="0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf"
Nov 03 15:41:37.986841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.986795    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf"} err="failed to get container status \"0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf\": rpc error: code = NotFound desc = could not find container \"0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf\": container with ID starting with 0390863b90679037d0878f200b25a65ebabb4268e385a6019ba1cbe77ebc6fdf not found: ID does not exist"
Nov 03 15:41:37.986841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.986815    4955 scope.go:122] "RemoveContainer" containerID="c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"
Nov 03 15:41:37.987114 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:37.987097    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616\": container with ID starting with c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616 not found: ID does not exist" containerID="c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"
Nov 03 15:41:37.987177 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:37.987118    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616"} err="failed to get container status \"c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616\": rpc error: code = NotFound desc = could not find container \"c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616\": container with ID starting with c4f7bc85cf53aad5d8de9125caeb8134c5cf5bc7728630b46175a7dc6aa34616 not found: ID does not exist"
Nov 03 15:41:38.137803 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:38.137769    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"]
Nov 03 15:41:38.186232 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:38.186195    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-7b3fd3-openshift-default-d9688497c-d4cvz"]
Nov 03 15:41:39.207670 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:39.207624    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" path="/var/lib/kubelet/pods/691ab01b-5d2e-4dd9-868e-c49f850e2e67/volumes"
Nov 03 15:41:52.228178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.228063    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 15:41:52.237121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.237064    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.238568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.238534    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.239564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.239435    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 03 15:41:52.459130 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.459077    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.479681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.479583    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.697996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.696538    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.713135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.713081    4955 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.907311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.906373    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:52.943001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:52.942586    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.109996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.109113    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.109996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.109512    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.236054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.231443    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.238748 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.238710    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.252028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.250153    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.254867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.254841    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:41:53.255139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.255123    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:41:53.265136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.265115    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:41:53.480395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.480351    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.507079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.506263    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:41:53.619606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.619569    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:41:53.705555 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.705511    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt" podUID="958b5803-9348-4904-ac21-82254f5f9b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:41:53.710104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.706366    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:41:53.799633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.799549    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:41:53.899686 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.899637    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt"
Nov 03 15:41:53.990026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.965458    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:41:53.990026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:53.971281    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:41:54.012992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:54.009918    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:41:54.097422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:54.097340    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:41:54.107729 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:41:54.107705    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:41:57.940519 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:41:57.940481    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:42:01.234693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:01.234660    4955 generic.go:356] "Generic (PLEG): container finished" podID="caab1123-8037-409b-af2e-31de277de6c8" containerID="592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee" exitCode=0
Nov 03 15:42:01.235159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:01.234700    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" event={"ID":"caab1123-8037-409b-af2e-31de277de6c8","Type":"ContainerDied","Data":"592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"}
Nov 03 15:42:01.235159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:01.235110    4955 scope.go:122] "RemoveContainer" containerID="592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"
Nov 03 15:42:01.236702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:01.236670    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:01.236791 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:01.236757    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:01.236826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:01.236791    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:02.262617 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:02.262578    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" event={"ID":"caab1123-8037-409b-af2e-31de277de6c8","Type":"ContainerStarted","Data":"bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9"}
Nov 03 15:42:02.389026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:02.388996    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:42:03.027194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:03.027165    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:42:03.028515 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:03.028489    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 15:42:03.028635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:03.028543    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 15:42:04.026755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:04.026720    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 15:42:04.027186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:04.026776    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 15:42:05.026565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.026522    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 15:42:05.026757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.026578    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 15:42:05.713346 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.713311    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"]
Nov 03 15:42:05.713934 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.713919    4955 cpu_manager.go:414] "containerMap: removing container" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy"
Nov 03 15:42:05.713934 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.713935    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy"
Nov 03 15:42:05.714089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.714078    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy"
Nov 03 15:42:05.714134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.714097    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy"
Nov 03 15:42:05.717750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.717735    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.776012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.772946    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-75edeb-serving-cert\""
Nov 03 15:42:05.802523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.802487    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.802718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.802539    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.802800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.802709    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52stl\" (UniqueName: \"kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.802860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.802810    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.826093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.826061    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"]
Nov 03 15:42:05.904218 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.904185    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.904389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.904227    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.904389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.904307    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52stl\" (UniqueName: \"kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.904389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.904358    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.904389 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:42:05.904360    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-serving-cert: secret "maas-api-e2e-shared-b-w1-75edeb-serving-cert" not found
Nov 03 15:42:05.904578 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:42:05.904441    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls podName:d95bec2b-299d-4046-bbdb-96d9783f8880 nodeName:}" failed. No retries permitted until 2026-11-03 15:42:06.404421251 +0000 UTC m=+2041.329239163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls") pod "maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" (UID: "d95bec2b-299d-4046-bbdb-96d9783f8880") : secret "maas-api-e2e-shared-b-w1-75edeb-serving-cert" not found
Nov 03 15:42:05.904913 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.904888    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.907103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.907081    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:05.976641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:05.976608    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52stl\" (UniqueName: \"kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:06.034989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:06.029025    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 15:42:06.034989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:06.029112    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 15:42:06.414420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:06.414386    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:06.417094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:06.417072    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:06.647079 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:06.645637    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:07.026556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:07.026468    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 15:42:07.026556 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:07.026528    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 15:42:07.056107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:07.056078    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"]
Nov 03 15:42:07.058957 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:42:07.058924    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95bec2b_299d_4046_bbdb_96d9783f8880.slice/crio-c08a953e3b3dd4500b8589626ab6c56e8519d3a9988994ed4e6e419ceb54b3a5 WatchSource:0}: Error finding container c08a953e3b3dd4500b8589626ab6c56e8519d3a9988994ed4e6e419ceb54b3a5: Status 404 returned error can't find the container with id c08a953e3b3dd4500b8589626ab6c56e8519d3a9988994ed4e6e419ceb54b3a5
Nov 03 15:42:07.297196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:07.297107    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" event={"ID":"d95bec2b-299d-4046-bbdb-96d9783f8880","Type":"ContainerStarted","Data":"c08a953e3b3dd4500b8589626ab6c56e8519d3a9988994ed4e6e419ceb54b3a5"}
Nov 03 15:42:08.054764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:08.051019    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:42:08.061398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:08.061369    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:42:08.306401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:08.306362    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" event={"ID":"d95bec2b-299d-4046-bbdb-96d9783f8880","Type":"ContainerStarted","Data":"d9e4ca5db6af9393815ff8d393ae7c8d9fc812f0451e7a6635702e27a99a6a9b"}
Nov 03 15:42:08.306630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:08.306607    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:14.318200 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:14.318170    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:42:14.448549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:14.448494    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" podStartSLOduration=8.655013896 podStartE2EDuration="9.448479495s" totalImagesPullingTime="793.465599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:08.491454588 +0000 UTC m=+2043.416272501" watchObservedRunningTime="2026-11-03 15:42:14.448479495 +0000 UTC m=+2049.373297463"
Nov 03 15:42:26.701611 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:26.701574    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:42:27.940452 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:42:27.940401    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:42:37.368177 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.368139    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:42:37.370511 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.368227    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:42:37.600593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.600525    4955 generic.go:356] "Generic (PLEG): container finished" podID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerID="c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51" exitCode=0
Nov 03 15:42:37.600738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.600600    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" event={"ID":"f909bbbf-ca76-48ac-9e32-4aa4d5613c67","Type":"ContainerDied","Data":"c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"}
Nov 03 15:42:37.601082 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.601064    4955 scope.go:122] "RemoveContainer" containerID="c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"
Nov 03 15:42:37.602798 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.602771    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:37.602856 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.602837    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:37.602894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:37.602870    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:38.608379 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:38.608344    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" event={"ID":"f909bbbf-ca76-48ac-9e32-4aa4d5613c67","Type":"ContainerStarted","Data":"66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1"}
Nov 03 15:42:39.201366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:39.201334    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 15:42:39.201562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:39.201391    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 15:42:39.206484 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:39.206460    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:42:40.201017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:40.200946    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 15:42:40.201441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:40.201069    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 15:42:41.212433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.212390    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 15:42:41.213127 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.213090    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 15:42:41.261183 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.261148    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"]
Nov 03 15:42:41.261720 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.261680    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" podUID="7eddbea6-7716-4cef-89cf-e6f67baaa593" containerName="maas-api" containerID="cri-o://963372608fdd0f995576340afc678da91204212f1a9f36f519415f7e063c4c5a" gracePeriod=30
Nov 03 15:42:41.693483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.693439    4955 generic.go:356] "Generic (PLEG): container finished" podID="7eddbea6-7716-4cef-89cf-e6f67baaa593" containerID="963372608fdd0f995576340afc678da91204212f1a9f36f519415f7e063c4c5a" exitCode=0
Nov 03 15:42:41.693834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.693811    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" event={"ID":"7eddbea6-7716-4cef-89cf-e6f67baaa593","Type":"ContainerDied","Data":"963372608fdd0f995576340afc678da91204212f1a9f36f519415f7e063c4c5a"}
Nov 03 15:42:41.952995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.947304    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:42:41.971735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.971701    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca\") pod \"7eddbea6-7716-4cef-89cf-e6f67baaa593\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") "
Nov 03 15:42:41.971926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.971743    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb\" (UniqueName: \"kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb\") pod \"7eddbea6-7716-4cef-89cf-e6f67baaa593\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") "
Nov 03 15:42:41.971926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.971794    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs\") pod \"7eddbea6-7716-4cef-89cf-e6f67baaa593\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") "
Nov 03 15:42:41.971926 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.971926    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\") pod \"7eddbea6-7716-4cef-89cf-e6f67baaa593\" (UID: \"7eddbea6-7716-4cef-89cf-e6f67baaa593\") "
Nov 03 15:42:41.972159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.972032    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca" pod "7eddbea6-7716-4cef-89cf-e6f67baaa593" (UID: "7eddbea6-7716-4cef-89cf-e6f67baaa593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:42:41.972584 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.972516    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eddbea6-7716-4cef-89cf-e6f67baaa593-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:42:41.974227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.974192    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls" pod "7eddbea6-7716-4cef-89cf-e6f67baaa593" (UID: "7eddbea6-7716-4cef-89cf-e6f67baaa593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:42:41.974316 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.974241    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb" pod "7eddbea6-7716-4cef-89cf-e6f67baaa593" (UID: "7eddbea6-7716-4cef-89cf-e6f67baaa593"). InnerVolumeSpecName "kube-api-access-hmpvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:42:41.974316 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:41.974272    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs" pod "7eddbea6-7716-4cef-89cf-e6f67baaa593" (UID: "7eddbea6-7716-4cef-89cf-e6f67baaa593"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:42:42.073761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.073713    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmpvb\" (UniqueName: \"kubernetes.io/projected/7eddbea6-7716-4cef-89cf-e6f67baaa593-kube-api-access-hmpvb\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:42:42.073761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.073756    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:42:42.073959 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.073772    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eddbea6-7716-4cef-89cf-e6f67baaa593-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:42:42.214054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.209092    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 15:42:42.214054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.209161    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 15:42:42.706756 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.706709    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7" event={"ID":"7eddbea6-7716-4cef-89cf-e6f67baaa593","Type":"ContainerDied","Data":"0c4fe6ae25afe4dc4e41ac1f7813b7a8ce5b3fd768a60bdbdffcc7ad0d72bb7a"}
Nov 03 15:42:42.707007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.706788    4955 scope.go:122] "RemoveContainer" containerID="963372608fdd0f995576340afc678da91204212f1a9f36f519415f7e063c4c5a"
Nov 03 15:42:42.707007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.706957    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"
Nov 03 15:42:42.928356 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.928316    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"]
Nov 03 15:42:42.966672 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:42.966574    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5cad072f-7f977db8fd-wkqt7"]
Nov 03 15:42:43.202750 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:43.202718    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 15:42:43.203045 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:43.203013    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 15:42:43.211310 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:43.211283    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eddbea6-7716-4cef-89cf-e6f67baaa593" path="/var/lib/kubelet/pods/7eddbea6-7716-4cef-89cf-e6f67baaa593/volumes"
Nov 03 15:42:44.287340 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:44.287292    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:42:44.287789 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:44.287635    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:42:44.288428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:44.288411    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:42:45.985652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.985608    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"]
Nov 03 15:42:45.987247 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.987218    4955 cpu_manager.go:414] "containerMap: removing container" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy"
Nov 03 15:42:45.987247 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.987244    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691ab01b-5d2e-4dd9-868e-c49f850e2e67" containerName="istio-proxy"
Nov 03 15:42:45.987441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.987349    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7eddbea6-7716-4cef-89cf-e6f67baaa593" containerName="maas-api"
Nov 03 15:42:45.987441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.987357    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eddbea6-7716-4cef-89cf-e6f67baaa593" containerName="maas-api"
Nov 03 15:42:45.987687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.987671    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eddbea6-7716-4cef-89cf-e6f67baaa593" containerName="maas-api"
Nov 03 15:42:45.994066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:45.994042    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083056    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083126    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvlp\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083193    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083396 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083350    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083445    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083473    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083493    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083586 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083575    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.083625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.083603    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.090757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.090733    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"]
Nov 03 15:42:46.185387 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185354    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185418    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvlp\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185546    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185716 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185698    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185829 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185815    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185844    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185877 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185854    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.185994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.185904    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.186062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.186026    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.186062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.186037    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.186166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.186092    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.186242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.186224    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.186447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.186424    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.187043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.187014    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.188139 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.188118    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.188450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.188434    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.249367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.249286    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.249542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.249471    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvlp\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.308929 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.308894    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:46.757451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.757409    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"]
Nov 03 15:42:46.758249 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:42:46.758221    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4734a279_01a0_4d25_a81d_3d15cc9634da.slice/crio-267a919b3001e89a51aeddf4c16f80fb945d8e7c40b365d2c7d3b292a8a2a9da WatchSource:0}: Error finding container 267a919b3001e89a51aeddf4c16f80fb945d8e7c40b365d2c7d3b292a8a2a9da: Status 404 returned error can't find the container with id 267a919b3001e89a51aeddf4c16f80fb945d8e7c40b365d2c7d3b292a8a2a9da
Nov 03 15:42:46.761176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.761145    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:46.761267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.761210    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:46.761267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:46.761238    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:42:47.769108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:47.769069    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" event={"ID":"4734a279-01a0-4d25-a81d-3d15cc9634da","Type":"ContainerStarted","Data":"a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3"}
Nov 03 15:42:47.769558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:47.769128    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" event={"ID":"4734a279-01a0-4d25-a81d-3d15cc9634da","Type":"ContainerStarted","Data":"267a919b3001e89a51aeddf4c16f80fb945d8e7c40b365d2c7d3b292a8a2a9da"}
Nov 03 15:42:47.992780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:47.992727    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podStartSLOduration=2.992711847 podStartE2EDuration="2.992711847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:42:47.991205368 +0000 UTC m=+2082.916023282" watchObservedRunningTime="2026-11-03 15:42:47.992711847 +0000 UTC m=+2082.917529839"
Nov 03 15:42:48.309060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:48.309027    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:48.310776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:48.310749    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 15:42:48.310880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:48.310793    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 15:42:49.309782 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:49.309745    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 15:42:49.310255 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:49.309800    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 15:42:50.309791 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:50.309757    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 15:42:50.310236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:50.309815    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 15:42:51.309933 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:51.309892    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 15:42:51.310361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:51.309949    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 15:42:52.310187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:52.310155    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 15:42:52.310609 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:52.310207    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 15:42:53.314577 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:53.314546    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:53.315035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:53.315012    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:53.315743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:42:53.315724    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:42:57.939630 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:42:57.939581    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:43:01.088637 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.088605    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"]
Nov 03 15:43:01.093201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.093160    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.175887 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.175851    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.175887 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.175887    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.175914    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.176056    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58lq\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176198 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.176182    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.176226    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176396 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.176384    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.176435    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.176525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.176508    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.196768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.196736    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"]
Nov 03 15:43:01.278110 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278072    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278110 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278115    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278246    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278280    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278339 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278310    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278348    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58lq\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278413    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278464    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278503 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278492    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278551    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278664    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278711 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278687    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.278962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.278857    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.279710 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.279684    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.280856 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.280837    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.281012 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.280996    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.349620 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.349551    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58lq\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.349620 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.349553    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token\") pod \"e2e-api-b-90fcca-openshift-default-575d955f5-642nc\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") " pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.408501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.408469    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:01.844583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.844557    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"]
Nov 03 15:43:01.845166 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:43:01.845142    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425b176f_1f9d_449c_8340_9a862828ca56.slice/crio-7ffc257d3d101382ff0bb70580be9518ead6b35cda75f5e2b69c7cb0aae66653 WatchSource:0}: Error finding container 7ffc257d3d101382ff0bb70580be9518ead6b35cda75f5e2b69c7cb0aae66653: Status 404 returned error can't find the container with id 7ffc257d3d101382ff0bb70580be9518ead6b35cda75f5e2b69c7cb0aae66653
Nov 03 15:43:01.848161 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.848128    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:43:01.848229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.848194    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:43:01.848229 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.848223    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:43:01.960871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.960839    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" event={"ID":"425b176f-1f9d-449c-8340-9a862828ca56","Type":"ContainerStarted","Data":"fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808"}
Nov 03 15:43:01.961081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:01.960888    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" event={"ID":"425b176f-1f9d-449c-8340-9a862828ca56","Type":"ContainerStarted","Data":"7ffc257d3d101382ff0bb70580be9518ead6b35cda75f5e2b69c7cb0aae66653"}
Nov 03 15:43:02.089758 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:02.089701    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podStartSLOduration=1.089685957 podStartE2EDuration="1.089685957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:43:02.086606426 +0000 UTC m=+2097.011424360" watchObservedRunningTime="2026-11-03 15:43:02.089685957 +0000 UTC m=+2097.014503869"
Nov 03 15:43:02.409457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:02.409418    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:02.410844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:02.410823    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:02.410910 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:02.410871    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:03.409154 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:03.409117    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:03.409553 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:03.409174    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:04.409226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:04.409186    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:04.409685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:04.409244    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:05.409194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:05.409157    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:05.409420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:05.409224    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:05.777562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:05.777465    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:43:05.786452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:05.786428    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:43:05.789634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:05.789616    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:43:05.797422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:05.797404    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:43:06.409828 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:06.409793    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:06.410226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:06.409850    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:07.083029 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:07.082984    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:43:07.409283 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:07.409246    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:07.409477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:07.409320    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:08.416509 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:08.416472    4955 patch_prober.go:29] interesting pod/e2e-api-b-90fcca-openshift-default-575d955f5-642nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 15:43:08.417156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:08.417119    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 15:43:09.113243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:09.113201    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:43:09.611428 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:09.611396    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:09.614723 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:09.613995    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:09.626411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:09.626242    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:43:09.776096 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:09.776054    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 03 15:43:10.013992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:10.013848    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:43:27.939706 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:43:27.939666    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:43:37.793950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:37.793914    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"]
Nov 03 15:43:37.794674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:37.794152    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" containerID="cri-o://537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf" gracePeriod=30
Nov 03 15:43:38.033776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:38.033740    4955 patch_prober.go:29] interesting pod/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:43:38.033960 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:38.033798    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:43:43.074419 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.074345    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:43:43.228327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228287    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228350    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228421    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228534 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228493    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228561    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228597    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:43:43.228704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228634    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:43:43.228704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228654    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228711    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228741    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.228904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228787    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:43:43.228904 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228885    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n\") pod \"fdb83b4e-681e-4545-b608-a6927912848f\" (UID: \"fdb83b4e-681e-4545-b608-a6927912848f\") "
Nov 03 15:43:43.229152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228938    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:43:43.229152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.228942    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:43:43.229448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.229429    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.229536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.229453    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.229536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.229467    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.229536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.229480    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb83b4e-681e-4545-b608-a6927912848f-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.229536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.229494    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.231100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.231075    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:43:43.231208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.231078    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:43:43.231433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.231415    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:43:43.231523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.231503    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n" pod "fdb83b4e-681e-4545-b608-a6927912848f" (UID: "fdb83b4e-681e-4545-b608-a6927912848f"). InnerVolumeSpecName "kube-api-access-dst5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:43:43.331194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.331111    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb83b4e-681e-4545-b608-a6927912848f-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.331194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.331140    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.331194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.331149    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dst5n\" (UniqueName: \"kubernetes.io/projected/fdb83b4e-681e-4545-b608-a6927912848f-kube-api-access-dst5n\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.331194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.331158    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb83b4e-681e-4545-b608-a6927912848f-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:43:43.504615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.504580    4955 generic.go:356] "Generic (PLEG): container finished" podID="fdb83b4e-681e-4545-b608-a6927912848f" containerID="537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf" exitCode=0
Nov 03 15:43:43.504615 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.504618    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"
Nov 03 15:43:43.504835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.504664    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" event={"ID":"fdb83b4e-681e-4545-b608-a6927912848f","Type":"ContainerDied","Data":"537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf"}
Nov 03 15:43:43.504835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.504720    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b" event={"ID":"fdb83b4e-681e-4545-b608-a6927912848f","Type":"ContainerDied","Data":"75770071f95dc6353d22ef2658f84d26182838cba2a113fe584bcb0aa2f7b9b4"}
Nov 03 15:43:43.504835 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.504745    4955 scope.go:122] "RemoveContainer" containerID="537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf"
Nov 03 15:43:43.515945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.515926    4955 scope.go:122] "RemoveContainer" containerID="537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf"
Nov 03 15:43:43.516312 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:43:43.516292    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf\": container with ID starting with 537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf not found: ID does not exist" containerID="537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf"
Nov 03 15:43:43.516375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.516321    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf"} err="failed to get container status \"537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf\": rpc error: code = NotFound desc = could not find container \"537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf\": container with ID starting with 537348df0a3253de2215e1b416eb50cc7049a39e083bc9599f09116b343127cf not found: ID does not exist"
Nov 03 15:43:43.679249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.679218    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"]
Nov 03 15:43:43.683623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:43.683594    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-2nc7b"]
Nov 03 15:43:45.206571 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:43:45.206534    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb83b4e-681e-4545-b608-a6927912848f" path="/var/lib/kubelet/pods/fdb83b4e-681e-4545-b608-a6927912848f/volumes"
Nov 03 15:43:57.939194 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:43:57.939143    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:44:13.396375 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:13.396336    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:44:15.605541 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:15.605504    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"]
Nov 03 15:44:15.606065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:15.605798    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy" containerID="cri-o://7ead3a9f2aa693c1fd25fe1c8b5019e0a93c3967c15b8b44eb1ebbab20933806" gracePeriod=30
Nov 03 15:44:15.616236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:15.616207    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"]
Nov 03 15:44:15.616464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:15.616418    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy" containerID="cri-o://64b4e621db491592b8557ce01d63ca8cd973e0c5a5092e50c7cb2b98a7afb32b" gracePeriod=30
Nov 03 15:44:20.878523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.878491    4955 generic.go:356] "Generic (PLEG): container finished" podID="994acb2b-546b-4ae3-bb39-bd580024257f" containerID="7ead3a9f2aa693c1fd25fe1c8b5019e0a93c3967c15b8b44eb1ebbab20933806" exitCode=0
Nov 03 15:44:20.878997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.878699    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" event={"ID":"994acb2b-546b-4ae3-bb39-bd580024257f","Type":"ContainerDied","Data":"7ead3a9f2aa693c1fd25fe1c8b5019e0a93c3967c15b8b44eb1ebbab20933806"}
Nov 03 15:44:20.878997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.878764    4955 scope.go:122] "RemoveContainer" containerID="633edd754680d0f86d75f0f8a6f1391ba16fb0e0f3a37afeaa6c501110888a32"
Nov 03 15:44:20.881490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.881459    4955 generic.go:356] "Generic (PLEG): container finished" podID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerID="64b4e621db491592b8557ce01d63ca8cd973e0c5a5092e50c7cb2b98a7afb32b" exitCode=0
Nov 03 15:44:20.881644 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.881624    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" event={"ID":"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5","Type":"ContainerDied","Data":"64b4e621db491592b8557ce01d63ca8cd973e0c5a5092e50c7cb2b98a7afb32b"}
Nov 03 15:44:20.915074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.915048    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:44:20.948386 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:20.948362    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.075787    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.075860    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.075891    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.075936    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076006    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076045    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076129    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076174    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076205    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076352    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076398    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076459    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076544    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076614    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076676    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076713    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076805    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data\") pod \"994acb2b-546b-4ae3-bb39-bd580024257f\" (UID: \"994acb2b-546b-4ae3-bb39-bd580024257f\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.076850    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data\") pod \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\" (UID: \"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5\") "
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.078017    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.078579    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.078914    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.079542    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.079871    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.080396    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.080586    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.080762    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.084993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.080927    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.091725 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.091684    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.092020 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.091942    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.093295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.093264    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:21.093410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.093324    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:44:21.093547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.093518    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "kube-api-access-z2mwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:21.093616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.093542    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:21.093666 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.093624    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:21.093734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.093710    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v" pod "994acb2b-546b-4ae3-bb39-bd580024257f" (UID: "994acb2b-546b-4ae3-bb39-bd580024257f"). InnerVolumeSpecName "kube-api-access-znj5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:21.095155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.095128    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo" pod "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" (UID: "ef8fce7d-b2e5-4eba-b73f-a5397b6830b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:44:21.179451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179414    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znj5v\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-kube-api-access-znj5v\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179446    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179451 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179457    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179466    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994acb2b-546b-4ae3-bb39-bd580024257f-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179476    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179484    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179492    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2mwv\" (UniqueName: \"kubernetes.io/projected/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-kube-api-access-z2mwv\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179501    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179509    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179517    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179524    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179532    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179539    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994acb2b-546b-4ae3-bb39-bd580024257f-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179547    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179555    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179563    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994acb2b-546b-4ae3-bb39-bd580024257f-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179571    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.179734 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.179578    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994acb2b-546b-4ae3-bb39-bd580024257f-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:21.889775 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.889679    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"
Nov 03 15:44:21.889775 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.889714    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r" event={"ID":"ef8fce7d-b2e5-4eba-b73f-a5397b6830b5","Type":"ContainerDied","Data":"602f0eb008831a139670763aa2dcad671d7ba692633848b78001384699b7d968"}
Nov 03 15:44:21.890354 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.889785    4955 scope.go:122] "RemoveContainer" containerID="64b4e621db491592b8557ce01d63ca8cd973e0c5a5092e50c7cb2b98a7afb32b"
Nov 03 15:44:21.891384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.891365    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"
Nov 03 15:44:21.891384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.891380    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz" event={"ID":"994acb2b-546b-4ae3-bb39-bd580024257f","Type":"ContainerDied","Data":"c58c770b6e264e5bcb26797d0bb905106a6598ef702bdced627916543c8503d2"}
Nov 03 15:44:21.900258 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:21.900242    4955 scope.go:122] "RemoveContainer" containerID="7ead3a9f2aa693c1fd25fe1c8b5019e0a93c3967c15b8b44eb1ebbab20933806"
Nov 03 15:44:22.059565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:22.059530    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"]
Nov 03 15:44:22.111656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:22.111621    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-5cgvz"]
Nov 03 15:44:22.177119 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:22.177078    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"]
Nov 03 15:44:22.183866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:22.183837    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-34e8c5-openshift-default-8547d97b6c-xql2r"]
Nov 03 15:44:23.220790 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:23.220755    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" path="/var/lib/kubelet/pods/994acb2b-546b-4ae3-bb39-bd580024257f/volumes"
Nov 03 15:44:23.221569 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:23.221550    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" path="/var/lib/kubelet/pods/ef8fce7d-b2e5-4eba-b73f-a5397b6830b5/volumes"
Nov 03 15:44:27.939880 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:27.939846    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:44:32.281496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:32.280640    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"]
Nov 03 15:44:32.281496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:32.281006    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" containerID="cri-o://e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9" gracePeriod=30
Nov 03 15:44:33.643674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:33.643636    4955 patch_prober.go:29] interesting pod/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:44:33.644104 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:33.643697    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:44:36.830399 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:36.830359    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"]
Nov 03 15:44:36.943641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:36.943603    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"]
Nov 03 15:44:37.584151 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.584089    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:44:37.632761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.632729    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.632909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.632823    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.632946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.632911    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.632954    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633021    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633067 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633059    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633183 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633116    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633244 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633215    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633240    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:37.633305 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633243    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:37.633414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633308    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket\") pod \"2ab2ea66-3da5-4b85-979f-a40a2253b523\" (UID: \"2ab2ea66-3da5-4b85-979f-a40a2253b523\") "
Nov 03 15:44:37.633414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633370    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:37.633574 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633551    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:37.633685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633665    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:37.633945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633922    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2ea66-3da5-4b85-979f-a40a2253b523-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.633945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633947    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.634155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633961    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.634155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.633996    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.634155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.634008    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.635320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.635295    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "kube-api-access-nww2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:37.635512 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.635495    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:37.635889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.635870    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:44:37.635953 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.635886    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy" pod "2ab2ea66-3da5-4b85-979f-a40a2253b523" (UID: "2ab2ea66-3da5-4b85-979f-a40a2253b523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:44:37.735751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.735706    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.735751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.735737    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.735751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.735746    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2ea66-3da5-4b85-979f-a40a2253b523-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:37.735751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:37.735760    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nww2c\" (UniqueName: \"kubernetes.io/projected/2ab2ea66-3da5-4b85-979f-a40a2253b523-kube-api-access-nww2c\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:38.005635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.005591    4955 generic.go:356] "Generic (PLEG): container finished" podID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerID="e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9" exitCode=0
Nov 03 15:44:38.006209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.005666    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"
Nov 03 15:44:38.006209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.005676    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" event={"ID":"2ab2ea66-3da5-4b85-979f-a40a2253b523","Type":"ContainerDied","Data":"e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9"}
Nov 03 15:44:38.006209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.005788    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7" event={"ID":"2ab2ea66-3da5-4b85-979f-a40a2253b523","Type":"ContainerDied","Data":"ef16e2e1f663bece8c07b217f0b6603fee9d8a265d5086890b8595ca8293e0d4"}
Nov 03 15:44:38.006209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.005822    4955 scope.go:122] "RemoveContainer" containerID="e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9"
Nov 03 15:44:38.019938 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.019904    4955 scope.go:122] "RemoveContainer" containerID="66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"
Nov 03 15:44:38.030132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.030111    4955 scope.go:122] "RemoveContainer" containerID="e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9"
Nov 03 15:44:38.030414 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:38.030388    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9\": container with ID starting with e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9 not found: ID does not exist" containerID="e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9"
Nov 03 15:44:38.030497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.030428    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9"} err="failed to get container status \"e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9\": rpc error: code = NotFound desc = could not find container \"e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9\": container with ID starting with e56503cfbfe060344dd79262e6c0d3f66f43cf236471713c52255beef30774f9 not found: ID does not exist"
Nov 03 15:44:38.030497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.030453    4955 scope.go:122] "RemoveContainer" containerID="66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"
Nov 03 15:44:38.030698 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:38.030679    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57\": container with ID starting with 66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57 not found: ID does not exist" containerID="66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"
Nov 03 15:44:38.030759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.030704    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57"} err="failed to get container status \"66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57\": rpc error: code = NotFound desc = could not find container \"66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57\": container with ID starting with 66ca2d2a839cd186e005b91d69ca2a947d0dcd1742ebc17aa28465d7064cbd57 not found: ID does not exist"
Nov 03 15:44:38.177722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.177687    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"]
Nov 03 15:44:38.228691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:38.228664    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-771ea0-openshift-default-6cc8bdf77f-7tfz7"]
Nov 03 15:44:39.206431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:39.206398    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" path="/var/lib/kubelet/pods/2ab2ea66-3da5-4b85-979f-a40a2253b523/volumes"
Nov 03 15:44:41.831469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:41.831410    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr" podUID="ab3f60b7-c7e7-4570-86fe-7c815204c860" containerName="payload-processing" containerID="cri-o://621bd84c9f07c2db1a8ae42f53abb0784ddb633fd272c12d2f93b1eb148f80dd" gracePeriod=25
Nov 03 15:44:41.944434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:41.944394    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg" podUID="58df5973-7bdc-4d61-bf48-da056f56bdf9" containerName="payload-pre-processing" containerID="cri-o://4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192" gracePeriod=25
Nov 03 15:44:42.036986 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.036932    4955 generic.go:356] "Generic (PLEG): container finished" podID="ab3f60b7-c7e7-4570-86fe-7c815204c860" containerID="621bd84c9f07c2db1a8ae42f53abb0784ddb633fd272c12d2f93b1eb148f80dd" exitCode=0
Nov 03 15:44:42.037156 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.037006    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr" event={"ID":"ab3f60b7-c7e7-4570-86fe-7c815204c860","Type":"ContainerDied","Data":"621bd84c9f07c2db1a8ae42f53abb0784ddb633fd272c12d2f93b1eb148f80dd"}
Nov 03 15:44:42.122807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.122784    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:44:42.206243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.206199    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume\") pod \"ab3f60b7-c7e7-4570-86fe-7c815204c860\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") "
Nov 03 15:44:42.206404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.206261    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr\" (UniqueName: \"kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr\") pod \"ab3f60b7-c7e7-4570-86fe-7c815204c860\" (UID: \"ab3f60b7-c7e7-4570-86fe-7c815204c860\") "
Nov 03 15:44:42.206613 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.206579    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume" pod "ab3f60b7-c7e7-4570-86fe-7c815204c860" (UID: "ab3f60b7-c7e7-4570-86fe-7c815204c860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:42.206805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.206771    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3f60b7-c7e7-4570-86fe-7c815204c860-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:42.208527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.208503    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr" pod "ab3f60b7-c7e7-4570-86fe-7c815204c860" (UID: "ab3f60b7-c7e7-4570-86fe-7c815204c860"). InnerVolumeSpecName "kube-api-access-6hnsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:42.227726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.227693    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:44:42.307457 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.307428    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume\") pod \"58df5973-7bdc-4d61-bf48-da056f56bdf9\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") "
Nov 03 15:44:42.307661 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.307467    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64\" (UniqueName: \"kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64\") pod \"58df5973-7bdc-4d61-bf48-da056f56bdf9\" (UID: \"58df5973-7bdc-4d61-bf48-da056f56bdf9\") "
Nov 03 15:44:42.307852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.307820    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume" pod "58df5973-7bdc-4d61-bf48-da056f56bdf9" (UID: "58df5973-7bdc-4d61-bf48-da056f56bdf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:42.308102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.308084    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58df5973-7bdc-4d61-bf48-da056f56bdf9-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:42.308154 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.308106    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnsr\" (UniqueName: \"kubernetes.io/projected/ab3f60b7-c7e7-4570-86fe-7c815204c860-kube-api-access-6hnsr\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:42.309817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.309796    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64" pod "58df5973-7bdc-4d61-bf48-da056f56bdf9" (UID: "58df5973-7bdc-4d61-bf48-da056f56bdf9"). InnerVolumeSpecName "kube-api-access-v2t64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:42.409434 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:42.409402    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2t64\" (UniqueName: \"kubernetes.io/projected/58df5973-7bdc-4d61-bf48-da056f56bdf9-kube-api-access-v2t64\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:43.045135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.045107    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"
Nov 03 15:44:43.045616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.045143    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr" event={"ID":"ab3f60b7-c7e7-4570-86fe-7c815204c860","Type":"ContainerDied","Data":"99801ab5bb781fd4d897166cd3be940b7824de785da91763bfd2dd3a8696b3f6"}
Nov 03 15:44:43.045616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.045211    4955 scope.go:122] "RemoveContainer" containerID="621bd84c9f07c2db1a8ae42f53abb0784ddb633fd272c12d2f93b1eb148f80dd"
Nov 03 15:44:43.046692 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.046663    4955 generic.go:356] "Generic (PLEG): container finished" podID="58df5973-7bdc-4d61-bf48-da056f56bdf9" containerID="4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192" exitCode=0
Nov 03 15:44:43.046826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.046695    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg" event={"ID":"58df5973-7bdc-4d61-bf48-da056f56bdf9","Type":"ContainerDied","Data":"4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192"}
Nov 03 15:44:43.046826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.046748    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg" event={"ID":"58df5973-7bdc-4d61-bf48-da056f56bdf9","Type":"ContainerDied","Data":"75a6aa52d423da10bc884e65212b3a5a507605f45336d02ef4ea4fddd325ca9d"}
Nov 03 15:44:43.046826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.046709    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"
Nov 03 15:44:43.056607 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.056590    4955 scope.go:122] "RemoveContainer" containerID="4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192"
Nov 03 15:44:43.068463 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.068367    4955 scope.go:122] "RemoveContainer" containerID="4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192"
Nov 03 15:44:43.068750 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:43.068732    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192\": container with ID starting with 4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192 not found: ID does not exist" containerID="4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192"
Nov 03 15:44:43.068802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.068760    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192"} err="failed to get container status \"4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192\": rpc error: code = NotFound desc = could not find container \"4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192\": container with ID starting with 4d14644612bc562c58371ada47481db1d788f7c03213f960873e27cb4e44a192 not found: ID does not exist"
Nov 03 15:44:43.219798 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.219757    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"]
Nov 03 15:44:43.225676 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.225649    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-38ded0-6879d5bc89-k7rmr"]
Nov 03 15:44:43.341865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.341830    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"]
Nov 03 15:44:43.348762 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:43.348732    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-38ded0-7dc59f7979-pghxg"]
Nov 03 15:44:44.649862 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:44.649780    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"]
Nov 03 15:44:44.766659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:44.766623    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"]
Nov 03 15:44:45.225057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:45.224789    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58df5973-7bdc-4d61-bf48-da056f56bdf9" path="/var/lib/kubelet/pods/58df5973-7bdc-4d61-bf48-da056f56bdf9/volumes"
Nov 03 15:44:45.226008 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:45.225950    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3f60b7-c7e7-4570-86fe-7c815204c860" path="/var/lib/kubelet/pods/ab3f60b7-c7e7-4570-86fe-7c815204c860/volumes"
Nov 03 15:44:49.651026 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:49.650953    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5" podUID="7062e4ce-9e99-4133-b1c7-3abee02a3e53" containerName="payload-processing" containerID="cri-o://46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4" gracePeriod=25
Nov 03 15:44:49.767307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:49.767254    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6" podUID="01b4f369-9910-4569-81f3-3cd7866ec52d" containerName="payload-pre-processing" containerID="cri-o://c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8" gracePeriod=25
Nov 03 15:44:49.932317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:49.932280    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:44:50.033485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.033457    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h\" (UniqueName: \"kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h\") pod \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") "
Nov 03 15:44:50.033671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.033582    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume\") pod \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\" (UID: \"7062e4ce-9e99-4133-b1c7-3abee02a3e53\") "
Nov 03 15:44:50.034023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.033983    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume" pod "7062e4ce-9e99-4133-b1c7-3abee02a3e53" (UID: "7062e4ce-9e99-4133-b1c7-3abee02a3e53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:50.034190 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.034173    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7062e4ce-9e99-4133-b1c7-3abee02a3e53-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:50.035727 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.035705    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h" pod "7062e4ce-9e99-4133-b1c7-3abee02a3e53" (UID: "7062e4ce-9e99-4133-b1c7-3abee02a3e53"). InnerVolumeSpecName "kube-api-access-br44h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:50.059714 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.059687    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:44:50.099200 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.099164    4955 generic.go:356] "Generic (PLEG): container finished" podID="7062e4ce-9e99-4133-b1c7-3abee02a3e53" containerID="46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4" exitCode=0
Nov 03 15:44:50.099388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.099228    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"
Nov 03 15:44:50.099388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.099308    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5" event={"ID":"7062e4ce-9e99-4133-b1c7-3abee02a3e53","Type":"ContainerDied","Data":"46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4"}
Nov 03 15:44:50.099388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.099349    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5" event={"ID":"7062e4ce-9e99-4133-b1c7-3abee02a3e53","Type":"ContainerDied","Data":"cd4515a8730405ea9ff71689d4825d899b5d035228df6bb5e043e6d988fb41d5"}
Nov 03 15:44:50.099388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.099375    4955 scope.go:122] "RemoveContainer" containerID="46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4"
Nov 03 15:44:50.101770 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.101742    4955 generic.go:356] "Generic (PLEG): container finished" podID="01b4f369-9910-4569-81f3-3cd7866ec52d" containerID="c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8" exitCode=0
Nov 03 15:44:50.101888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.101815    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6" event={"ID":"01b4f369-9910-4569-81f3-3cd7866ec52d","Type":"ContainerDied","Data":"c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8"}
Nov 03 15:44:50.101888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.101859    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6" event={"ID":"01b4f369-9910-4569-81f3-3cd7866ec52d","Type":"ContainerDied","Data":"95c3ee073655d74560594ee904fa626ec0867170289899cb5cb3b4741fcb58d2"}
Nov 03 15:44:50.102022 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.101945    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"
Nov 03 15:44:50.111237 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.111213    4955 scope.go:122] "RemoveContainer" containerID="46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4"
Nov 03 15:44:50.111565 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:50.111544    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4\": container with ID starting with 46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4 not found: ID does not exist" containerID="46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4"
Nov 03 15:44:50.111663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.111574    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4"} err="failed to get container status \"46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4\": rpc error: code = NotFound desc = could not find container \"46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4\": container with ID starting with 46e6f0f5a3f6b550f864a6e3534b1bf572b050a1788f7a10fb1d03aca33d7cc4 not found: ID does not exist"
Nov 03 15:44:50.111663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.111596    4955 scope.go:122] "RemoveContainer" containerID="c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8"
Nov 03 15:44:50.120833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.120813    4955 scope.go:122] "RemoveContainer" containerID="c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8"
Nov 03 15:44:50.121142 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:50.121122    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8\": container with ID starting with c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8 not found: ID does not exist" containerID="c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8"
Nov 03 15:44:50.121210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.121152    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8"} err="failed to get container status \"c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8\": rpc error: code = NotFound desc = could not find container \"c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8\": container with ID starting with c5934572a14a56fa48df4e79aa03b0207ba19059301648dca91204a0ae557ee8 not found: ID does not exist"
Nov 03 15:44:50.135585 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.135561    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume\") pod \"01b4f369-9910-4569-81f3-3cd7866ec52d\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") "
Nov 03 15:44:50.135679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.135667    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd\" (UniqueName: \"kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd\") pod \"01b4f369-9910-4569-81f3-3cd7866ec52d\" (UID: \"01b4f369-9910-4569-81f3-3cd7866ec52d\") "
Nov 03 15:44:50.135949 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.135929    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume" pod "01b4f369-9910-4569-81f3-3cd7866ec52d" (UID: "01b4f369-9910-4569-81f3-3cd7866ec52d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:50.136186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.136170    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b4f369-9910-4569-81f3-3cd7866ec52d-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:50.136227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.136190    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br44h\" (UniqueName: \"kubernetes.io/projected/7062e4ce-9e99-4133-b1c7-3abee02a3e53-kube-api-access-br44h\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:50.137778 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.137759    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd" pod "01b4f369-9910-4569-81f3-3cd7866ec52d" (UID: "01b4f369-9910-4569-81f3-3cd7866ec52d"). InnerVolumeSpecName "kube-api-access-kf7wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:50.237395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.237317    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7wd\" (UniqueName: \"kubernetes.io/projected/01b4f369-9910-4569-81f3-3cd7866ec52d-kube-api-access-kf7wd\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:50.294497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.294461    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"]
Nov 03 15:44:50.299362 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.299332    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-75edeb-669f8bc84f-ds6s5"]
Nov 03 15:44:50.528536 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.528441    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"]
Nov 03 15:44:50.531634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.531607    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-75edeb-f54df47bc-mxmk6"]
Nov 03 15:44:50.916225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.916191    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"]
Nov 03 15:44:50.916985 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.916946    4955 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.916986    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917005    4955 cpu_manager.go:414] "containerMap: removing container" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917012    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917022    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7062e4ce-9e99-4133-b1c7-3abee02a3e53" containerName="payload-processing"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917030    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7062e4ce-9e99-4133-b1c7-3abee02a3e53" containerName="payload-processing"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917067    4955 cpu_manager.go:414] "containerMap: removing container" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy"
Nov 03 15:44:50.917090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917075    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917106    4955 cpu_manager.go:414] "containerMap: removing container" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917114    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917127    4955 cpu_manager.go:414] "containerMap: removing container" podUID="ab3f60b7-c7e7-4570-86fe-7c815204c860" containerName="payload-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917134    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3f60b7-c7e7-4570-86fe-7c815204c860" containerName="payload-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917145    4955 cpu_manager.go:414] "containerMap: removing container" podUID="01b4f369-9910-4569-81f3-3cd7866ec52d" containerName="payload-pre-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917152    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b4f369-9910-4569-81f3-3cd7866ec52d" containerName="payload-pre-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917174    4955 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917181    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917199    4955 cpu_manager.go:414] "containerMap: removing container" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917205    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917221    4955 cpu_manager.go:414] "containerMap: removing container" podUID="58df5973-7bdc-4d61-bf48-da056f56bdf9" containerName="payload-pre-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917228    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58df5973-7bdc-4d61-bf48-da056f56bdf9" containerName="payload-pre-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917368    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917381    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="58df5973-7bdc-4d61-bf48-da056f56bdf9" containerName="payload-pre-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917392    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917402    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8fce7d-b2e5-4eba-b73f-a5397b6830b5" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917410    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7062e4ce-9e99-4133-b1c7-3abee02a3e53" containerName="payload-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917423    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb83b4e-681e-4545-b608-a6927912848f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917436    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="994acb2b-546b-4ae3-bb39-bd580024257f" containerName="istio-proxy"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917445    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3f60b7-c7e7-4570-86fe-7c815204c860" containerName="payload-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917453    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b4f369-9910-4569-81f3-3cd7866ec52d" containerName="payload-pre-processing"
Nov 03 15:44:50.917488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.917466    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2ea66-3da5-4b85-979f-a40a2253b523" containerName="istio-proxy"
Nov 03 15:44:50.922705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.922683    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:50.935560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.935526    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"]
Nov 03 15:44:50.939605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:50.939583    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.028773 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.028740    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"]
Nov 03 15:44:51.047273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.047239    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"]
Nov 03 15:44:51.053932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.053886    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.053985    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054023    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054049    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054092    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054120    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054145    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054183    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054268    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054310    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054369    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054446    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv26c\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054472    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgw7c\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054506    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054532    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.054702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054569    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.054702 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054613    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.055037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.054719    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156164 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156129    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156224    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156308    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156372    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156413 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156399    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156425    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156466    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156493    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156517    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156556    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156593    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156648    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156708    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156704    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156810    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv26c\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156842    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgw7c\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156878    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156902    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156902    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156596    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.156962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.156940    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.157547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157036    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.157547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157054    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.157547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157229    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.157547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157295    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.157547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157311    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.157547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157534    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.157948 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.157930    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.159143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.159116    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.159459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.159442    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.159792 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.159778    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.160246 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.160231    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.206377 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.206278    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b4f369-9910-4569-81f3-3cd7866ec52d" path="/var/lib/kubelet/pods/01b4f369-9910-4569-81f3-3cd7866ec52d/volumes"
Nov 03 15:44:51.206839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.206822    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7062e4ce-9e99-4133-b1c7-3abee02a3e53" path="/var/lib/kubelet/pods/7062e4ce-9e99-4133-b1c7-3abee02a3e53/volumes"
Nov 03 15:44:51.227531 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.227472    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.228188 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.228165    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgw7c\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.230191 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.230164    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv26c\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.230324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.230304    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.238852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.238832    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:51.253688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.253667    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:51.289470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.289434    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"]
Nov 03 15:44:51.401468 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.401347    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"]
Nov 03 15:44:51.680460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.680429    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"]
Nov 03 15:44:51.681251 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:44:51.681225    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5534d9_f720_4262_a098_c5c8d9b3ff9e.slice/crio-9a536f0376e1810d0a3bcc39836151d2bd8fda9e2927ae6ccebc94a8d512bbc3 WatchSource:0}: Error finding container 9a536f0376e1810d0a3bcc39836151d2bd8fda9e2927ae6ccebc94a8d512bbc3: Status 404 returned error can't find the container with id 9a536f0376e1810d0a3bcc39836151d2bd8fda9e2927ae6ccebc94a8d512bbc3
Nov 03 15:44:51.684135 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.684093    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:44:51.684225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.684182    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:44:51.684266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.684225    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:44:51.761422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.761291    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"]
Nov 03 15:44:51.790599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.790331    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:44:51.790599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.790407    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:44:51.790599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:51.790439    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:44:52.128806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.128697    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" event={"ID":"3b5534d9-f720-4262-a098-c5c8d9b3ff9e","Type":"ContainerStarted","Data":"0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d"}
Nov 03 15:44:52.128806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.128768    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" event={"ID":"3b5534d9-f720-4262-a098-c5c8d9b3ff9e","Type":"ContainerStarted","Data":"9a536f0376e1810d0a3bcc39836151d2bd8fda9e2927ae6ccebc94a8d512bbc3"}
Nov 03 15:44:52.131590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.131563    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" event={"ID":"d463324a-0f30-4de2-b492-77adaf702b09","Type":"ContainerStarted","Data":"02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962"}
Nov 03 15:44:52.131800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.131606    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" event={"ID":"d463324a-0f30-4de2-b492-77adaf702b09","Type":"ContainerStarted","Data":"2b790eb96b3afe3b155f3adef5b5a3fdc69ddd166a47c8eea102c30367ee640f"}
Nov 03 15:44:52.239787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.239750    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:52.241326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.241299    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:52.241469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.241344    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:52.253755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.253694    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podStartSLOduration=2.253675695 podStartE2EDuration="2.253675695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:52.252137568 +0000 UTC m=+2207.176955507" watchObservedRunningTime="2026-11-03 15:44:52.253675695 +0000 UTC m=+2207.178493608"
Nov 03 15:44:52.254634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.254535    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:52.256187 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.256164    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:52.256309 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.256219    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:52.384705 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:52.384577    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podStartSLOduration=2.384557275 podStartE2EDuration="2.384557275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:44:52.383655577 +0000 UTC m=+2207.308473491" watchObservedRunningTime="2026-11-03 15:44:52.384557275 +0000 UTC m=+2207.309375190"
Nov 03 15:44:53.266996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:53.258865    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:53.266996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:53.258937    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:53.266996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:53.263118    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:53.266996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:53.263172    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:54.260999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:54.250393    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:54.260999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:54.250454    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:54.269009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:54.263318    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:54.269009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:54.263380    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:55.255001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:55.242356    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:55.255001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:55.242420    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:55.263007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:55.261380    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:55.263007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:55.261448    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:56.249997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.241356    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:56.249997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.241418    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:56.254993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.254847    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:56.255176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.255005    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:56.290361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.290316    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw" podUID="44cc575e-ced1-4e36-a805-6b7d3982fe7b" containerName="payload-processing" containerID="cri-o://297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a" gracePeriod=25
Nov 03 15:44:56.401948 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.401901    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472" podUID="53c8fe04-de0d-40d4-9cf4-2a8294138819" containerName="payload-pre-processing" containerID="cri-o://0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35" gracePeriod=25
Nov 03 15:44:56.723112 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.723087    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:44:56.819775 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.819698    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc\") pod \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") "
Nov 03 15:44:56.820151 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.820135    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume\") pod \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\" (UID: \"44cc575e-ced1-4e36-a805-6b7d3982fe7b\") "
Nov 03 15:44:56.820787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.820765    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume" pod "44cc575e-ced1-4e36-a805-6b7d3982fe7b" (UID: "44cc575e-ced1-4e36-a805-6b7d3982fe7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:56.824497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.824470    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc" pod "44cc575e-ced1-4e36-a805-6b7d3982fe7b" (UID: "44cc575e-ced1-4e36-a805-6b7d3982fe7b"). InnerVolumeSpecName "kube-api-access-gjrhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:56.900023 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.899991    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:44:56.923016 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.922945    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrhc\" (UniqueName: \"kubernetes.io/projected/44cc575e-ced1-4e36-a805-6b7d3982fe7b-kube-api-access-gjrhc\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:56.923160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:56.923027    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cc575e-ced1-4e36-a805-6b7d3982fe7b-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:57.024519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.024485    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume\") pod \"53c8fe04-de0d-40d4-9cf4-2a8294138819\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") "
Nov 03 15:44:57.024820 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.024803    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr\" (UniqueName: \"kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr\") pod \"53c8fe04-de0d-40d4-9cf4-2a8294138819\" (UID: \"53c8fe04-de0d-40d4-9cf4-2a8294138819\") "
Nov 03 15:44:57.024945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.024893    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume" pod "53c8fe04-de0d-40d4-9cf4-2a8294138819" (UID: "53c8fe04-de0d-40d4-9cf4-2a8294138819"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:44:57.025299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.025279    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8fe04-de0d-40d4-9cf4-2a8294138819-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:57.027124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.027105    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr" pod "53c8fe04-de0d-40d4-9cf4-2a8294138819" (UID: "53c8fe04-de0d-40d4-9cf4-2a8294138819"). InnerVolumeSpecName "kube-api-access-x6wvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:44:57.126205 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.126130    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6wvr\" (UniqueName: \"kubernetes.io/projected/53c8fe04-de0d-40d4-9cf4-2a8294138819-kube-api-access-x6wvr\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:44:57.206683 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.206637    4955 generic.go:356] "Generic (PLEG): container finished" podID="44cc575e-ced1-4e36-a805-6b7d3982fe7b" containerID="297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a" exitCode=0
Nov 03 15:44:57.206863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.206841    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"
Nov 03 15:44:57.207275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.207243    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw" event={"ID":"44cc575e-ced1-4e36-a805-6b7d3982fe7b","Type":"ContainerDied","Data":"297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a"}
Nov 03 15:44:57.207397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.207301    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw" event={"ID":"44cc575e-ced1-4e36-a805-6b7d3982fe7b","Type":"ContainerDied","Data":"2a29a73c5981f985f2978d7ee223a78f2b9cd8f13c05d2959b11daeef166129a"}
Nov 03 15:44:57.207397 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.207333    4955 scope.go:122] "RemoveContainer" containerID="297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a"
Nov 03 15:44:57.208837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.208810    4955 generic.go:356] "Generic (PLEG): container finished" podID="53c8fe04-de0d-40d4-9cf4-2a8294138819" containerID="0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35" exitCode=0
Nov 03 15:44:57.208917 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.208855    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"
Nov 03 15:44:57.209017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.208997    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472" event={"ID":"53c8fe04-de0d-40d4-9cf4-2a8294138819","Type":"ContainerDied","Data":"0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35"}
Nov 03 15:44:57.209062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.209045    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472" event={"ID":"53c8fe04-de0d-40d4-9cf4-2a8294138819","Type":"ContainerDied","Data":"0b2744ce08e99cce8068a3acdf0765ade113d2f23ecd163fc57f61929786e10f"}
Nov 03 15:44:57.221903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.221205    4955 scope.go:122] "RemoveContainer" containerID="297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a"
Nov 03 15:44:57.221903 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:57.221564    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a\": container with ID starting with 297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a not found: ID does not exist" containerID="297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a"
Nov 03 15:44:57.221903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.221594    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a"} err="failed to get container status \"297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a\": rpc error: code = NotFound desc = could not find container \"297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a\": container with ID starting with 297bd2d9ca99fdc23994d32f9a25f831a2f30228d8a9762ed8e203062775a91a not found: ID does not exist"
Nov 03 15:44:57.221903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.221612    4955 scope.go:122] "RemoveContainer" containerID="0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35"
Nov 03 15:44:57.252997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.239378    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:57.252997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.239436    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:57.279526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.279489    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:57.279717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.279549    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:57.279717 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.279604    4955 scope.go:122] "RemoveContainer" containerID="0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35"
Nov 03 15:44:57.280236 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:57.280184    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35\": container with ID starting with 0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35 not found: ID does not exist" containerID="0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35"
Nov 03 15:44:57.280236 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.280215    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35"} err="failed to get container status \"0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35\": rpc error: code = NotFound desc = could not find container \"0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35\": container with ID starting with 0010e25ca69329f31838459fb3092bd9338fe0fe27a73a4fc3cb488887686f35 not found: ID does not exist"
Nov 03 15:44:57.392125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.392097    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"]
Nov 03 15:44:57.397360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.397335    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fe0cb9-5fdfc85c87-7p472"]
Nov 03 15:44:57.557996 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.557043    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"]
Nov 03 15:44:57.579997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:57.566845    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fe0cb9-dcbc4594c-ptbpw"]
Nov 03 15:44:57.982814 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:44:57.982730    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:44:58.239447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:58.239340    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 15:44:58.239447 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:58.239417    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 15:44:58.254827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:58.254794    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 03 15:44:58.255215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:58.254849    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 03 15:44:59.205827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.205791    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cc575e-ced1-4e36-a805-6b7d3982fe7b" path="/var/lib/kubelet/pods/44cc575e-ced1-4e36-a805-6b7d3982fe7b/volumes"
Nov 03 15:44:59.206267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.206252    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c8fe04-de0d-40d4-9cf4-2a8294138819" path="/var/lib/kubelet/pods/53c8fe04-de0d-40d4-9cf4-2a8294138819/volumes"
Nov 03 15:44:59.243788 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.243759    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:59.244100 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.244077    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:59.244891 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.244876    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:44:59.259487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.259454    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:59.259906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.259755    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:44:59.260659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:44:59.260643    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:45:00.239799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.239762    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"]
Nov 03 15:45:00.240786 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.240758    4955 cpu_manager.go:414] "containerMap: removing container" podUID="44cc575e-ced1-4e36-a805-6b7d3982fe7b" containerName="payload-processing"
Nov 03 15:45:00.240999 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.240944    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cc575e-ced1-4e36-a805-6b7d3982fe7b" containerName="payload-processing"
Nov 03 15:45:00.241089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.241049    4955 cpu_manager.go:414] "containerMap: removing container" podUID="53c8fe04-de0d-40d4-9cf4-2a8294138819" containerName="payload-pre-processing"
Nov 03 15:45:00.241089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.241062    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c8fe04-de0d-40d4-9cf4-2a8294138819" containerName="payload-pre-processing"
Nov 03 15:45:00.241259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.241244    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c8fe04-de0d-40d4-9cf4-2a8294138819" containerName="payload-pre-processing"
Nov 03 15:45:00.241320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.241274    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cc575e-ced1-4e36-a805-6b7d3982fe7b" containerName="payload-processing"
Nov 03 15:45:00.246251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.246217    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.324049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.324007    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"]
Nov 03 15:45:00.356769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.356740    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\""
Nov 03 15:45:00.427667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.427633    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.427873 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.427694    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzf2\" (UniqueName: \"kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.529596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.529512    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.529596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.529555    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzf2\" (UniqueName: \"kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.530212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.530188    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.591952 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.591918    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"]
Nov 03 15:45:00.592252 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.592223    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" containerID="cri-o://6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a" gracePeriod=30
Nov 03 15:45:00.593228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.593194    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzf2\" (UniqueName: \"kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2\") pod \"maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:00.863506 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:00.863415    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:01.258514 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:01.258484    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"]
Nov 03 15:45:01.258890 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:45:01.258868    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247f7ba1_a6ff_4d7d_9411_8f1c7311777e.slice/crio-8c5fa37e5fc66709c080b8cf9d933bbbef3ce927e69f845aeafd65aec7bb697a WatchSource:0}: Error finding container 8c5fa37e5fc66709c080b8cf9d933bbbef3ce927e69f845aeafd65aec7bb697a: Status 404 returned error can't find the container with id 8c5fa37e5fc66709c080b8cf9d933bbbef3ce927e69f845aeafd65aec7bb697a
Nov 03 15:45:02.262342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:02.262307    4955 generic.go:356] "Generic (PLEG): container finished" podID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerID="80d908f4b197b9b60b525be7b833ebbbbe21cabbb7ed0c1bd3e1bc4b73cccac5" exitCode=60
Nov 03 15:45:02.262776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:02.262388    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" event={"ID":"247f7ba1-a6ff-4d7d-9411-8f1c7311777e","Type":"ContainerDied","Data":"80d908f4b197b9b60b525be7b833ebbbbe21cabbb7ed0c1bd3e1bc4b73cccac5"}
Nov 03 15:45:02.262776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:02.262481    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" event={"ID":"247f7ba1-a6ff-4d7d-9411-8f1c7311777e","Type":"ContainerStarted","Data":"8c5fa37e5fc66709c080b8cf9d933bbbef3ce927e69f845aeafd65aec7bb697a"}
Nov 03 15:45:02.262776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:02.262582    4955 scope.go:122] "RemoveContainer" containerID="80d908f4b197b9b60b525be7b833ebbbbe21cabbb7ed0c1bd3e1bc4b73cccac5"
Nov 03 15:45:03.270889 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:03.270857    4955 generic.go:356] "Generic (PLEG): container finished" podID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerID="663c2b2ed4182ffd9a6df4c13d78bfb1f64fe3e36337af7d9e43fcab4bfdf8b1" exitCode=60
Nov 03 15:45:03.271395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:03.270915    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" event={"ID":"247f7ba1-a6ff-4d7d-9411-8f1c7311777e","Type":"ContainerDied","Data":"663c2b2ed4182ffd9a6df4c13d78bfb1f64fe3e36337af7d9e43fcab4bfdf8b1"}
Nov 03 15:45:03.271395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:03.271016    4955 scope.go:122] "RemoveContainer" containerID="80d908f4b197b9b60b525be7b833ebbbbe21cabbb7ed0c1bd3e1bc4b73cccac5"
Nov 03 15:45:03.271395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:03.271287    4955 scope.go:122] "RemoveContainer" containerID="663c2b2ed4182ffd9a6df4c13d78bfb1f64fe3e36337af7d9e43fcab4bfdf8b1"
Nov 03 15:45:03.271599 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:03.271574    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn_odh-ai-gateway-infra(247f7ba1-a6ff-4d7d-9411-8f1c7311777e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e"
Nov 03 15:45:04.279565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:04.279539    4955 scope.go:122] "RemoveContainer" containerID="663c2b2ed4182ffd9a6df4c13d78bfb1f64fe3e36337af7d9e43fcab4bfdf8b1"
Nov 03 15:45:04.279938 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:04.279856    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn_odh-ai-gateway-infra(247f7ba1-a6ff-4d7d-9411-8f1c7311777e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e"
Nov 03 15:45:04.991235 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:04.991191    4955 patch_prober.go:29] interesting pod/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:45:04.991388 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:04.991273    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:45:05.869812 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.869750    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:45:05.906336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906308    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906374    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906411    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906454    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906514    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906675    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906710    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906759    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.906923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906796    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:05.906923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906809    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:05.906923 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.906807    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert\") pod \"a019f010-05ff-47d9-8a42-bc507ab4780c\" (UID: \"a019f010-05ff-47d9-8a42-bc507ab4780c\") "
Nov 03 15:45:05.907093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907023    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:05.907299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907131    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:05.907299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907201    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:05.907771 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907744    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:05.907842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907777    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:05.907842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907793    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:05.907842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907807    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019f010-05ff-47d9-8a42-bc507ab4780c-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:05.907842 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.907821    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:05.909007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.908955    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:45:05.909101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.909018    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:05.909148 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.909129    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "kube-api-access-k2h74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:05.909267 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:05.909250    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token" pod "a019f010-05ff-47d9-8a42-bc507ab4780c" (UID: "a019f010-05ff-47d9-8a42-bc507ab4780c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:06.008591 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.008558    4955 scope.go:122] "RemoveContainer" containerID="12f052df44439d26c050b909271de51d590306cb2f1b9b2fee145379ae780440"
Nov 03 15:45:06.009627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.009598    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:06.009805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.009630    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2h74\" (UniqueName: \"kubernetes.io/projected/a019f010-05ff-47d9-8a42-bc507ab4780c-kube-api-access-k2h74\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:06.009805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.009646    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:06.009805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.009660    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019f010-05ff-47d9-8a42-bc507ab4780c-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.099293    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"]
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.100171    4955 cpu_manager.go:414] "containerMap: removing container" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy"
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.100188    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy"
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.100200    4955 cpu_manager.go:414] "containerMap: removing container" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy"
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.100208    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy"
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.100373    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy"
Nov 03 15:45:06.101013 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.100385    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerName="istio-proxy"
Nov 03 15:45:06.108414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.108391    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.128488 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.128459    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"]
Nov 03 15:45:06.215441 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215410    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215557    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215631 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215601    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215643    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5swp\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215717    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215752    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215768    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215890 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215784    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.215925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.215893    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.294479 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.294443    4955 generic.go:356] "Generic (PLEG): container finished" podID="a019f010-05ff-47d9-8a42-bc507ab4780c" containerID="6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a" exitCode=0
Nov 03 15:45:06.294676 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.294494    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"
Nov 03 15:45:06.294676 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.294522    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" event={"ID":"a019f010-05ff-47d9-8a42-bc507ab4780c","Type":"ContainerDied","Data":"6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a"}
Nov 03 15:45:06.294676 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.294576    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2" event={"ID":"a019f010-05ff-47d9-8a42-bc507ab4780c","Type":"ContainerDied","Data":"adb951aabc386b0b511390f50db4c49a4dc5912754cf80f66e0dd0fdc2d3ebdd"}
Nov 03 15:45:06.294676 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.294603    4955 scope.go:122] "RemoveContainer" containerID="6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a"
Nov 03 15:45:06.304861 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.304845    4955 scope.go:122] "RemoveContainer" containerID="6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a"
Nov 03 15:45:06.305205 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:06.305184    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a\": container with ID starting with 6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a not found: ID does not exist" containerID="6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a"
Nov 03 15:45:06.305266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.305216    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a"} err="failed to get container status \"6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a\": rpc error: code = NotFound desc = could not find container \"6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a\": container with ID starting with 6b37ceab3cc14b34c91c02734881acf5849c57ab04a555b41045db80a1a3384a not found: ID does not exist"
Nov 03 15:45:06.317578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317543    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317587    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317646    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5swp\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317731    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317769    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317797    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317822    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.317908 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.317863    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.318184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.318157    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.318264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.318200    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.318492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.318472    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.318659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.318639    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.318746 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.318726    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.318819 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.318796    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.320769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.320751    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.320822 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.320792    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.380357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.380285    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.380492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.380354    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5swp\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp\") pod \"e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") " pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.433840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.433810    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:06.544204 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.538752    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"]
Nov 03 15:45:06.546179 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.546146    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5d3cde-openshift-default-dfb468857-hr7h2"]
Nov 03 15:45:06.877932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.877903    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"]
Nov 03 15:45:06.879046 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:45:06.879020    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295300b9_1fc6_40af_a64e_fbf48c8c3bc6.slice/crio-b75e86d4e8517fbc9ff7df0fe1d70af252ffa3c96cd0defd7702796105034cfc WatchSource:0}: Error finding container b75e86d4e8517fbc9ff7df0fe1d70af252ffa3c96cd0defd7702796105034cfc: Status 404 returned error can't find the container with id b75e86d4e8517fbc9ff7df0fe1d70af252ffa3c96cd0defd7702796105034cfc
Nov 03 15:45:06.881743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.881711    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:45:06.881827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.881777    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:45:06.881827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:06.881805    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:45:07.209159 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.209124    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a019f010-05ff-47d9-8a42-bc507ab4780c" path="/var/lib/kubelet/pods/a019f010-05ff-47d9-8a42-bc507ab4780c/volumes"
Nov 03 15:45:07.354994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.343548    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" event={"ID":"295300b9-1fc6-40af-a64e-fbf48c8c3bc6","Type":"ContainerStarted","Data":"5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca"}
Nov 03 15:45:07.354994 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.343613    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" event={"ID":"295300b9-1fc6-40af-a64e-fbf48c8c3bc6","Type":"ContainerStarted","Data":"b75e86d4e8517fbc9ff7df0fe1d70af252ffa3c96cd0defd7702796105034cfc"}
Nov 03 15:45:07.436227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.436181    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 15:45:07.436497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.436453    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 15:45:07.436677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.436651    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:07.482945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:07.482820    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podStartSLOduration=1.482797892 podStartE2EDuration="1.482797892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:45:07.467606567 +0000 UTC m=+2222.392424481" watchObservedRunningTime="2026-11-03 15:45:07.482797892 +0000 UTC m=+2222.407615807"
Nov 03 15:45:08.417422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:08.417390    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:45:08.450995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:08.450359    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 15:45:08.450995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:08.450424    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 15:45:09.441998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:09.437405    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 15:45:09.441998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:09.437475    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 15:45:10.447408 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:10.447367    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 15:45:10.448065 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:10.448028    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 15:45:11.454390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:11.454176    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 15:45:11.454390 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:11.454247    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 15:45:12.434394 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:12.434355    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 03 15:45:12.434663 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:12.434632    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 03 15:45:13.518554 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:13.518521    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:13.520051 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:13.520024    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:13.522559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:13.522520    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:45:16.198606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:16.198572    4955 scope.go:122] "RemoveContainer" containerID="663c2b2ed4182ffd9a6df4c13d78bfb1f64fe3e36337af7d9e43fcab4bfdf8b1"
Nov 03 15:45:17.427380 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:17.427345    4955 generic.go:356] "Generic (PLEG): container finished" podID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerID="03fa5e4507f7640e65c64594d3ccbd4ef748822a0623c848d2e418f1d052ee8d" exitCode=60
Nov 03 15:45:17.427817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:17.427420    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" event={"ID":"247f7ba1-a6ff-4d7d-9411-8f1c7311777e","Type":"ContainerDied","Data":"03fa5e4507f7640e65c64594d3ccbd4ef748822a0623c848d2e418f1d052ee8d"}
Nov 03 15:45:17.427817 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:17.427478    4955 scope.go:122] "RemoveContainer" containerID="663c2b2ed4182ffd9a6df4c13d78bfb1f64fe3e36337af7d9e43fcab4bfdf8b1"
Nov 03 15:45:17.427939 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:17.427887    4955 scope.go:122] "RemoveContainer" containerID="03fa5e4507f7640e65c64594d3ccbd4ef748822a0623c848d2e418f1d052ee8d"
Nov 03 15:45:17.428251 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:17.428228    4955 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn_odh-ai-gateway-infra(247f7ba1-a6ff-4d7d-9411-8f1c7311777e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e"
Nov 03 15:45:18.437772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.437741    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"]
Nov 03 15:45:18.438206 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.437946    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" podUID="3c8117ac-048b-48fb-9316-90f39450be94" containerName="maas-api" containerID="cri-o://1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4" gracePeriod=30
Nov 03 15:45:18.612149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.584809    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"]
Nov 03 15:45:18.826444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.826417    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:45:18.967035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.966458    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca\") pod \"3c8117ac-048b-48fb-9316-90f39450be94\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") "
Nov 03 15:45:18.967035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.966515    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq\" (UniqueName: \"kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq\") pod \"3c8117ac-048b-48fb-9316-90f39450be94\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") "
Nov 03 15:45:18.967035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.966851    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs\") pod \"3c8117ac-048b-48fb-9316-90f39450be94\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") "
Nov 03 15:45:18.967035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.966866    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca" pod "3c8117ac-048b-48fb-9316-90f39450be94" (UID: "3c8117ac-048b-48fb-9316-90f39450be94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:18.967035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.966879    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\") pod \"3c8117ac-048b-48fb-9316-90f39450be94\" (UID: \"3c8117ac-048b-48fb-9316-90f39450be94\") "
Nov 03 15:45:18.967628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.967597    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8117ac-048b-48fb-9316-90f39450be94-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:18.981838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.981798    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls" pod "3c8117ac-048b-48fb-9316-90f39450be94" (UID: "3c8117ac-048b-48fb-9316-90f39450be94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:45:18.985932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.985103    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs" pod "3c8117ac-048b-48fb-9316-90f39450be94" (UID: "3c8117ac-048b-48fb-9316-90f39450be94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:45:18.985932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.985235    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq" pod "3c8117ac-048b-48fb-9316-90f39450be94" (UID: "3c8117ac-048b-48fb-9316-90f39450be94"). InnerVolumeSpecName "kube-api-access-p8ftq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:18.992081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:18.991760    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:19.068776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.068690    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca\") pod \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") "
Nov 03 15:45:19.068954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.068882    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2\" (UniqueName: \"kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2\") pod \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\" (UID: \"247f7ba1-a6ff-4d7d-9411-8f1c7311777e\") "
Nov 03 15:45:19.069090 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.069062    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca" pod "247f7ba1-a6ff-4d7d-9411-8f1c7311777e" (UID: "247f7ba1-a6ff-4d7d-9411-8f1c7311777e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:19.069526 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.069503    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:19.069657 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.069532    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8117ac-048b-48fb-9316-90f39450be94-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:19.069657 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.069550    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ftq\" (UniqueName: \"kubernetes.io/projected/3c8117ac-048b-48fb-9316-90f39450be94-kube-api-access-p8ftq\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:19.069657 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.069566    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-maas-api-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:19.071444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.071419    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2" pod "247f7ba1-a6ff-4d7d-9411-8f1c7311777e" (UID: "247f7ba1-a6ff-4d7d-9411-8f1c7311777e"). InnerVolumeSpecName "kube-api-access-9fzf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:19.180009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.176367    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzf2\" (UniqueName: \"kubernetes.io/projected/247f7ba1-a6ff-4d7d-9411-8f1c7311777e-kube-api-access-9fzf2\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:19.515992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.515420    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn" event={"ID":"247f7ba1-a6ff-4d7d-9411-8f1c7311777e","Type":"ContainerDied","Data":"8c5fa37e5fc66709c080b8cf9d933bbbef3ce927e69f845aeafd65aec7bb697a"}
Nov 03 15:45:19.515992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.515499    4955 scope.go:122] "RemoveContainer" containerID="03fa5e4507f7640e65c64594d3ccbd4ef748822a0623c848d2e418f1d052ee8d"
Nov 03 15:45:19.515992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.515655    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"
Nov 03 15:45:19.568043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.561239    4955 generic.go:356] "Generic (PLEG): container finished" podID="3c8117ac-048b-48fb-9316-90f39450be94" containerID="1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4" exitCode=0
Nov 03 15:45:19.568043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.561326    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" event={"ID":"3c8117ac-048b-48fb-9316-90f39450be94","Type":"ContainerDied","Data":"1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4"}
Nov 03 15:45:19.568043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.561475    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5" event={"ID":"3c8117ac-048b-48fb-9316-90f39450be94","Type":"ContainerDied","Data":"2ff4d4837f2418ec0bb8bb471a79be10c5c6c735d64d361edfd76cd6178c5f04"}
Nov 03 15:45:19.568043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.561345    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"
Nov 03 15:45:19.583338 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.582304    4955 scope.go:122] "RemoveContainer" containerID="1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4"
Nov 03 15:45:19.597616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.597584    4955 scope.go:122] "RemoveContainer" containerID="1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4"
Nov 03 15:45:19.598073 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:19.598045    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4\": container with ID starting with 1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4 not found: ID does not exist" containerID="1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4"
Nov 03 15:45:19.598196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.598090    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4"} err="failed to get container status \"1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4\": rpc error: code = NotFound desc = could not find container \"1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4\": container with ID starting with 1ebf0686c7105bfc5e90e37d68f76436668a26503c843a7c032b6728338039f4 not found: ID does not exist"
Nov 03 15:45:19.745720 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.745633    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"]
Nov 03 15:45:19.796083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.796051    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-38ded0-6985c8b55d-mxwh5"]
Nov 03 15:45:19.863287 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.863252    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"]
Nov 03 15:45:19.867678 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:19.867646    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9855ff-29895345-jh8fn"]
Nov 03 15:45:21.209685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:21.209653    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" path="/var/lib/kubelet/pods/247f7ba1-a6ff-4d7d-9411-8f1c7311777e/volumes"
Nov 03 15:45:21.210210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:21.210137    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8117ac-048b-48fb-9316-90f39450be94" path="/var/lib/kubelet/pods/3c8117ac-048b-48fb-9316-90f39450be94/volumes"
Nov 03 15:45:27.736424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:27.736388    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"]
Nov 03 15:45:27.736876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:27.736612    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy" containerID="cri-o://b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28" gracePeriod=30
Nov 03 15:45:27.978201 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:27.978149    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:45:33.026825 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.026795    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:45:33.107498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107453    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.107498 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107503    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.107764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107592    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.107764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107662    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.107764 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107702    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.107928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107805    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.107928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107842    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:33.107928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107851    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.108120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.107939    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.108120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.108069    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data\") pod \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\" (UID: \"168c6154-e58d-4ca2-86e3-4893f1b7d9ae\") "
Nov 03 15:45:33.108120 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.108084    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:33.108766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.108649    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.108766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.108677    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.109038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.108883    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:33.109137 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.109056    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:33.109323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.109306    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:33.110391 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.110367    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "kube-api-access-wfsbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:33.110499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.110385    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:45:33.110846 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.110827    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:33.111813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.111793    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token" pod "168c6154-e58d-4ca2-86e3-4893f1b7d9ae" (UID: "168c6154-e58d-4ca2-86e3-4893f1b7d9ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:33.209931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.209899    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.209931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.209929    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfsbr\" (UniqueName: \"kubernetes.io/projected/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-kube-api-access-wfsbr\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.209931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.209939    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.210264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.209949    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.210264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.209958    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.210264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.209985    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.210264 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.210001    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168c6154-e58d-4ca2-86e3-4893f1b7d9ae-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:33.675396 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.675360    4955 generic.go:356] "Generic (PLEG): container finished" podID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerID="b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28" exitCode=0
Nov 03 15:45:33.675578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.675408    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"
Nov 03 15:45:33.675578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.675440    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" event={"ID":"168c6154-e58d-4ca2-86e3-4893f1b7d9ae","Type":"ContainerDied","Data":"b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28"}
Nov 03 15:45:33.675578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.675501    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d" event={"ID":"168c6154-e58d-4ca2-86e3-4893f1b7d9ae","Type":"ContainerDied","Data":"a4d9c957ac3163b84fdbf3107ac5dafd77c7bb10ac601bc310395fafaa8c8c08"}
Nov 03 15:45:33.675578 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.675524    4955 scope.go:122] "RemoveContainer" containerID="b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28"
Nov 03 15:45:33.685429 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.685412    4955 scope.go:122] "RemoveContainer" containerID="19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"
Nov 03 15:45:33.694433 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.694412    4955 scope.go:122] "RemoveContainer" containerID="b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28"
Nov 03 15:45:33.694849 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:33.694789    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28\": container with ID starting with b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28 not found: ID does not exist" containerID="b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28"
Nov 03 15:45:33.695003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.694856    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28"} err="failed to get container status \"b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28\": rpc error: code = NotFound desc = could not find container \"b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28\": container with ID starting with b4b21c463bc490406b454c8bd073e868d66dc02cabc0abe27add008492d6ce28 not found: ID does not exist"
Nov 03 15:45:33.695003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.694878    4955 scope.go:122] "RemoveContainer" containerID="19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"
Nov 03 15:45:33.695214 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:33.695190    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021\": container with ID starting with 19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021 not found: ID does not exist" containerID="19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"
Nov 03 15:45:33.695260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.695224    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021"} err="failed to get container status \"19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021\": rpc error: code = NotFound desc = could not find container \"19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021\": container with ID starting with 19106f735b123b555c482084739153a39414b4de6de05a52b08a4978b2225021 not found: ID does not exist"
Nov 03 15:45:33.844024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.843991    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"]
Nov 03 15:45:33.894548 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:33.894509    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-38ded0-openshift-default-8cf996759-8cd7d"]
Nov 03 15:45:35.207638 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.207604    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" path="/var/lib/kubelet/pods/168c6154-e58d-4ca2-86e3-4893f1b7d9ae/volumes"
Nov 03 15:45:35.434931 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.434886    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"]
Nov 03 15:45:35.435151 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.435102    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" podUID="d95bec2b-299d-4046-bbdb-96d9783f8880" containerName="maas-api" containerID="cri-o://d9e4ca5db6af9393815ff8d393ae7c8d9fc812f0451e7a6635702e27a99a6a9b" gracePeriod=30
Nov 03 15:45:35.718132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.717704    4955 generic.go:356] "Generic (PLEG): container finished" podID="d95bec2b-299d-4046-bbdb-96d9783f8880" containerID="d9e4ca5db6af9393815ff8d393ae7c8d9fc812f0451e7a6635702e27a99a6a9b" exitCode=0
Nov 03 15:45:35.718132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.717928    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" event={"ID":"d95bec2b-299d-4046-bbdb-96d9783f8880","Type":"ContainerDied","Data":"d9e4ca5db6af9393815ff8d393ae7c8d9fc812f0451e7a6635702e27a99a6a9b"}
Nov 03 15:45:35.757450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.757426    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:45:35.846048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.845593    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs\") pod \"d95bec2b-299d-4046-bbdb-96d9783f8880\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") "
Nov 03 15:45:35.846048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.845717    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca\") pod \"d95bec2b-299d-4046-bbdb-96d9783f8880\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") "
Nov 03 15:45:35.846048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.845792    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl\" (UniqueName: \"kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl\") pod \"d95bec2b-299d-4046-bbdb-96d9783f8880\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") "
Nov 03 15:45:35.846048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.845890    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\") pod \"d95bec2b-299d-4046-bbdb-96d9783f8880\" (UID: \"d95bec2b-299d-4046-bbdb-96d9783f8880\") "
Nov 03 15:45:35.847459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.847428    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca" pod "d95bec2b-299d-4046-bbdb-96d9783f8880" (UID: "d95bec2b-299d-4046-bbdb-96d9783f8880"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:35.862216 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.862177    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls" pod "d95bec2b-299d-4046-bbdb-96d9783f8880" (UID: "d95bec2b-299d-4046-bbdb-96d9783f8880"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:45:35.905232 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.905194    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs" pod "d95bec2b-299d-4046-bbdb-96d9783f8880" (UID: "d95bec2b-299d-4046-bbdb-96d9783f8880"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:45:35.911056 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.910734    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl" pod "d95bec2b-299d-4046-bbdb-96d9783f8880" (UID: "d95bec2b-299d-4046-bbdb-96d9783f8880"). InnerVolumeSpecName "kube-api-access-52stl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:35.947564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.947528    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:35.947564 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.947563    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95bec2b-299d-4046-bbdb-96d9783f8880-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:35.947780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.947577    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52stl\" (UniqueName: \"kubernetes.io/projected/d95bec2b-299d-4046-bbdb-96d9783f8880-kube-api-access-52stl\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:35.947780 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:35.947591    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95bec2b-299d-4046-bbdb-96d9783f8880-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:36.632844 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.632814    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"]
Nov 03 15:45:36.633348 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.633036    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" podUID="8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" containerName="maas-api" containerID="cri-o://b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc" gracePeriod=30
Nov 03 15:45:36.728035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.728003    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j" event={"ID":"d95bec2b-299d-4046-bbdb-96d9783f8880","Type":"ContainerDied","Data":"c08a953e3b3dd4500b8589626ab6c56e8519d3a9988994ed4e6e419ceb54b3a5"}
Nov 03 15:45:36.728228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.728050    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"
Nov 03 15:45:36.728228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.728072    4955 scope.go:122] "RemoveContainer" containerID="d9e4ca5db6af9393815ff8d393ae7c8d9fc812f0451e7a6635702e27a99a6a9b"
Nov 03 15:45:36.924991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.919602    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"]
Nov 03 15:45:36.924991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.924880    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-75edeb-7685c6f8b-vwt2j"]
Nov 03 15:45:36.968231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:36.968201    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:45:37.047017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.034837    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\") pod \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") "
Nov 03 15:45:37.047017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.043105    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj\" (UniqueName: \"kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj\") pod \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") "
Nov 03 15:45:37.047017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.043235    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs\") pod \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") "
Nov 03 15:45:37.047017 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.043288    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca\") pod \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\" (UID: \"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e\") "
Nov 03 15:45:37.053037 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.052797    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj" pod "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" (UID: "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e"). InnerVolumeSpecName "kube-api-access-jmhgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:37.058342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.055477    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs" pod "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" (UID: "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:45:37.058342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.057081    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls" pod "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" (UID: "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:45:37.058342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.057263    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca" pod "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" (UID: "8f46fe3a-c661-4ce1-8a74-f80c501c3c2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:37.145317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.145282    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:37.145317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.145314    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmhgj\" (UniqueName: \"kubernetes.io/projected/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-kube-api-access-jmhgj\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:37.145317 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.145324    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:37.145639 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.145333    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:37.206709 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.206626    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95bec2b-299d-4046-bbdb-96d9783f8880" path="/var/lib/kubelet/pods/d95bec2b-299d-4046-bbdb-96d9783f8880/volumes"
Nov 03 15:45:37.735313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.735280    4955 generic.go:356] "Generic (PLEG): container finished" podID="8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" containerID="b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc" exitCode=0
Nov 03 15:45:37.735794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.735406    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"
Nov 03 15:45:37.735794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.735439    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" event={"ID":"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e","Type":"ContainerDied","Data":"b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc"}
Nov 03 15:45:37.735794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.735502    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp" event={"ID":"8f46fe3a-c661-4ce1-8a74-f80c501c3c2e","Type":"ContainerDied","Data":"2636e6c0942c2c2e33656b8cc31f0eb885244248b807cb29795ac6bf44c74b63"}
Nov 03 15:45:37.735794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.735534    4955 scope.go:122] "RemoveContainer" containerID="b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc"
Nov 03 15:45:37.755928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.755907    4955 scope.go:122] "RemoveContainer" containerID="b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc"
Nov 03 15:45:37.756311 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:37.756278    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc\": container with ID starting with b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc not found: ID does not exist" containerID="b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc"
Nov 03 15:45:37.756382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.756333    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc"} err="failed to get container status \"b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc\": rpc error: code = NotFound desc = could not find container \"b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc\": container with ID starting with b4ffd63052d849f162e4daad33db7524633ff2acda48fa1a89fc2b96360900dc not found: ID does not exist"
Nov 03 15:45:37.928658 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.928621    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"]
Nov 03 15:45:37.979125 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:37.979093    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe0cb9-65888cc58f-f7rdp"]
Nov 03 15:45:39.206381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:39.206352    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" path="/var/lib/kubelet/pods/8f46fe3a-c661-4ce1-8a74-f80c501c3c2e/volumes"
Nov 03 15:45:45.302462 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:45.302424    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"]
Nov 03 15:45:45.302893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:45.302629    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy" containerID="cri-o://66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1" gracePeriod=30
Nov 03 15:45:47.337485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:47.337444    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"]
Nov 03 15:45:47.338021 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:47.337660    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy" containerID="cri-o://81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0" gracePeriod=30
Nov 03 15:45:50.584372 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.584305    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:45:50.627006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.626952    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627026    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627100    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627343 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627330    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627408 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627383    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627466 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627426    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627466 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627454    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627486    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.627704 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.627559    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert\") pod \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\" (UID: \"f909bbbf-ca76-48ac-9e32-4aa4d5613c67\") "
Nov 03 15:45:50.628383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.628358    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:50.628602 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.628586    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:50.629111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.628738    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:50.629111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.628942    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:50.629259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.629172    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:50.630011 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.629950    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:50.630207 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.630181    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:50.632768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.632737    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:45:50.632768 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.632741    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl" pod "f909bbbf-ca76-48ac-9e32-4aa4d5613c67" (UID: "f909bbbf-ca76-48ac-9e32-4aa4d5613c67"). InnerVolumeSpecName "kube-api-access-6zdvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:50.729469 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729429    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729476    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zdvl\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-kube-api-access-6zdvl\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729492    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729507    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729521    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729535    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729548    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729563    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.729652 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.729577    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bbbf-ca76-48ac-9e32-4aa4d5613c67-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:50.763925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.763892    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"]
Nov 03 15:45:50.766914 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.766853    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"]
Nov 03 15:45:50.769909 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.769868    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"]
Nov 03 15:45:50.838718 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.838632    4955 generic.go:356] "Generic (PLEG): container finished" podID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerID="66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1" exitCode=0
Nov 03 15:45:50.838888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.838728    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" event={"ID":"f909bbbf-ca76-48ac-9e32-4aa4d5613c67","Type":"ContainerDied","Data":"66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1"}
Nov 03 15:45:50.838888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.838739    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"
Nov 03 15:45:50.838888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.838782    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j" event={"ID":"f909bbbf-ca76-48ac-9e32-4aa4d5613c67","Type":"ContainerDied","Data":"e44624cc64febcd05cae71f58ce8f3b866f3446ec3633eb15d4ca6406e7829b3"}
Nov 03 15:45:50.838888 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.838811    4955 scope.go:122] "RemoveContainer" containerID="66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1"
Nov 03 15:45:50.863145 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.863123    4955 scope.go:122] "RemoveContainer" containerID="c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"
Nov 03 15:45:50.872579 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.872554    4955 scope.go:122] "RemoveContainer" containerID="66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1"
Nov 03 15:45:50.872884 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:50.872865    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1\": container with ID starting with 66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1 not found: ID does not exist" containerID="66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1"
Nov 03 15:45:50.872942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.872893    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1"} err="failed to get container status \"66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1\": rpc error: code = NotFound desc = could not find container \"66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1\": container with ID starting with 66a33bf2fc4000c20e04ee53f0a441848bd155b3a63d74a004c380342b44b5f1 not found: ID does not exist"
Nov 03 15:45:50.872942 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.872912    4955 scope.go:122] "RemoveContainer" containerID="c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"
Nov 03 15:45:50.873187 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:50.873167    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51\": container with ID starting with c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51 not found: ID does not exist" containerID="c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"
Nov 03 15:45:50.873230 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:50.873196    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51"} err="failed to get container status \"c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51\": rpc error: code = NotFound desc = could not find container \"c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51\": container with ID starting with c798f0787b2e9e8aab1db9c66013f7f7fbf4282c182b06e93bfbb8c975ca1b51 not found: ID does not exist"
Nov 03 15:45:51.242901 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:51.242869    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"]
Nov 03 15:45:51.248184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:51.248156    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-75edeb-openshift-default-7c696db77f-wpn8j"]
Nov 03 15:45:52.599296 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.599267    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:45:52.652411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652366    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652417    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652445    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652597 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652508    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652647    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652719    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:52.652765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652742    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:52.652765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652747    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652789    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652853    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.652992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652879    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data\") pod \"4ba150f8-5538-4677-90c9-178795e8c449\" (UID: \"4ba150f8-5538-4677-90c9-178795e8c449\") "
Nov 03 15:45:52.653160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.652997    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:52.653361 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653333    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:52.653411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653358    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:52.653832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653770    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.653832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653801    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba150f8-5538-4677-90c9-178795e8c449-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.653832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653816    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.653832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653832    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.653832 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.653849    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.655028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.654999    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:52.655132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.655055    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:52.655257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.655234    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "kube-api-access-4pl96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:52.655444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.655426    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo" pod "4ba150f8-5538-4677-90c9-178795e8c449" (UID: "4ba150f8-5538-4677-90c9-178795e8c449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:45:52.755736 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.755695    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl96\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-kube-api-access-4pl96\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.755736 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.755737    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba150f8-5538-4677-90c9-178795e8c449-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.756007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.755755    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba150f8-5538-4677-90c9-178795e8c449-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.756007 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.755769    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba150f8-5538-4677-90c9-178795e8c449-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:52.857641 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.857606    4955 generic.go:356] "Generic (PLEG): container finished" podID="4ba150f8-5538-4677-90c9-178795e8c449" containerID="81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0" exitCode=0
Nov 03 15:45:52.857813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.857653    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"
Nov 03 15:45:52.857813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.857691    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" event={"ID":"4ba150f8-5538-4677-90c9-178795e8c449","Type":"ContainerDied","Data":"81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0"}
Nov 03 15:45:52.857813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.857749    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q" event={"ID":"4ba150f8-5538-4677-90c9-178795e8c449","Type":"ContainerDied","Data":"a7f4b8b44b5e0a2f4c2ad989c571bfb3bc80c37b96b6191e2e2cc4c42660b21e"}
Nov 03 15:45:52.857813 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.857771    4955 scope.go:122] "RemoveContainer" containerID="81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0"
Nov 03 15:45:52.868489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.868461    4955 scope.go:122] "RemoveContainer" containerID="d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"
Nov 03 15:45:52.878105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.878086    4955 scope.go:122] "RemoveContainer" containerID="81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0"
Nov 03 15:45:52.878406 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:52.878385    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0\": container with ID starting with 81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0 not found: ID does not exist" containerID="81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0"
Nov 03 15:45:52.878460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.878415    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0"} err="failed to get container status \"81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0\": rpc error: code = NotFound desc = could not find container \"81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0\": container with ID starting with 81c3df5cd955d63d49e5ef521928c1c5184d9b470fbd54fc33035d998d5461a0 not found: ID does not exist"
Nov 03 15:45:52.878460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.878435    4955 scope.go:122] "RemoveContainer" containerID="d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"
Nov 03 15:45:52.878714 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:52.878692    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051\": container with ID starting with d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051 not found: ID does not exist" containerID="d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"
Nov 03 15:45:52.878767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:52.878720    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051"} err="failed to get container status \"d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051\": rpc error: code = NotFound desc = could not find container \"d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051\": container with ID starting with d8fe1050bad2ba71e7ed399fc6102676ed0d26ed25ec3b0be1aeebb27dce5051 not found: ID does not exist"
Nov 03 15:45:53.026551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:53.026470    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"]
Nov 03 15:45:53.032265 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:53.032236    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fe0cb9-openshift-default-b8bb9855c-kl98q"]
Nov 03 15:45:53.206144 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:53.206111    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba150f8-5538-4677-90c9-178795e8c449" path="/var/lib/kubelet/pods/4ba150f8-5538-4677-90c9-178795e8c449/volumes"
Nov 03 15:45:53.206673 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:53.206657    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" path="/var/lib/kubelet/pods/f909bbbf-ca76-48ac-9e32-4aa4d5613c67/volumes"
Nov 03 15:45:53.827655 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:53.827621    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"]
Nov 03 15:45:53.828131 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:53.827823    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy" containerID="cri-o://39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8" gracePeriod=30
Nov 03 15:45:57.938887 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:57.938848    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:45:59.111438 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.111372    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:45:59.271321 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271283    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271362    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271542    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271605    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271628    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:59.271667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271646    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271697    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:45:59.271759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271714    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271779    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.271885 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271830    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.272010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.271952    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:59.272103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272049    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:59.272103 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272094    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket\") pod \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\" (UID: \"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1\") "
Nov 03 15:45:59.272351 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272332    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:59.272707 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272689    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.272783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272709    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.272783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272718    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.272783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272727    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.272783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.272736    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.274173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.274141    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:45:59.274290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.274190    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:59.274411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.274388    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "kube-api-access-ns9lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:45:59.274501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.274457    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo" pod "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" (UID: "2aa52f94-f41d-46fa-8b35-6b72cdbc05c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:45:59.314691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.314654    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:45:59.374102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.374036    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9lj\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-kube-api-access-ns9lj\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.374102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.374062    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.374102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.374072    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.374102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.374081    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:45:59.930995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.930874    4955 generic.go:356] "Generic (PLEG): container finished" podID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerID="39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8" exitCode=0
Nov 03 15:45:59.931210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.931089    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" event={"ID":"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1","Type":"ContainerDied","Data":"39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8"}
Nov 03 15:45:59.931210 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.931189    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"
Nov 03 15:45:59.931331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.931239    4955 scope.go:122] "RemoveContainer" containerID="39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8"
Nov 03 15:45:59.936189 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.933276    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh" event={"ID":"2aa52f94-f41d-46fa-8b35-6b72cdbc05c1","Type":"ContainerDied","Data":"78350a3bd1984df38f8861fa6ce5316aad545512941362b987f756efcf79ae38"}
Nov 03 15:45:59.960871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.960845    4955 scope.go:122] "RemoveContainer" containerID="868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"
Nov 03 15:45:59.987448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.987417    4955 scope.go:122] "RemoveContainer" containerID="39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8"
Nov 03 15:45:59.988657 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:59.988629    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8\": container with ID starting with 39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8 not found: ID does not exist" containerID="39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8"
Nov 03 15:45:59.988840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.988817    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8"} err="failed to get container status \"39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8\": rpc error: code = NotFound desc = could not find container \"39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8\": container with ID starting with 39775657fd71b4b2080365d641f13a51125d5c765493ce31733286eb3f0676e8 not found: ID does not exist"
Nov 03 15:45:59.988945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.988934    4955 scope.go:122] "RemoveContainer" containerID="868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"
Nov 03 15:45:59.989629 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:45:59.989607    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639\": container with ID starting with 868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639 not found: ID does not exist" containerID="868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"
Nov 03 15:45:59.989763 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:45:59.989741    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639"} err="failed to get container status \"868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639\": rpc error: code = NotFound desc = could not find container \"868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639\": container with ID starting with 868c48ebe1b98ab1478648bc613524942f0359d290ec869569fe19adee11c639 not found: ID does not exist"
Nov 03 15:46:00.114382 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:00.114343    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"]
Nov 03 15:46:00.169993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:00.169000    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bc1f07-openshift-default-796f66cf55-c54wh"]
Nov 03 15:46:00.621814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:00.621777    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"]
Nov 03 15:46:01.216826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:01.216788    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" path="/var/lib/kubelet/pods/2aa52f94-f41d-46fa-8b35-6b72cdbc05c1/volumes"
Nov 03 15:46:01.948093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:01.948052    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"]
Nov 03 15:46:05.622699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.622658    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk" podUID="1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" containerName="payload-processing" containerID="cri-o://70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008" gracePeriod=25
Nov 03 15:46:05.786445 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.786386    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" containerID="cri-o://a015048f1078b11b579f0442c97a480775308817c7e2bbba49e126c382b663e7" gracePeriod=45
Nov 03 15:46:05.790034 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.789998    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" containerID="cri-o://f72351ec3d6d4c7237740f50af41ba61b73f1068c0742375e17fffe04456ff3a" gracePeriod=45
Nov 03 15:46:05.791184 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.791129    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main" containerID="cri-o://7c6139d0d5969877638bee9eb0138033eaaac059bcc5bc2ec1bc9eaf4c11debc" gracePeriod=45
Nov 03 15:46:05.981470 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.980199    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:46:05.988510 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.988092    4955 generic.go:356] "Generic (PLEG): container finished" podID="59910387-1111-4c0b-a838-247b1c8173b3" containerID="7c6139d0d5969877638bee9eb0138033eaaac059bcc5bc2ec1bc9eaf4c11debc" exitCode=0
Nov 03 15:46:05.988510 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.988191    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" event={"ID":"59910387-1111-4c0b-a838-247b1c8173b3","Type":"ContainerDied","Data":"7c6139d0d5969877638bee9eb0138033eaaac059bcc5bc2ec1bc9eaf4c11debc"}
Nov 03 15:46:05.994286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.993807    4955 generic.go:356] "Generic (PLEG): container finished" podID="1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" containerID="70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008" exitCode=0
Nov 03 15:46:05.994286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.993920    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk" event={"ID":"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb","Type":"ContainerDied","Data":"70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008"}
Nov 03 15:46:05.994286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.993960    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk" event={"ID":"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb","Type":"ContainerDied","Data":"d8fffc99ea339aefe65ded049d969c55f3d436a1b677ab9776d59e03d4ece8fb"}
Nov 03 15:46:05.994286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.994026    4955 scope.go:122] "RemoveContainer" containerID="70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008"
Nov 03 15:46:05.994286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.994140    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"
Nov 03 15:46:05.996804 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.996587    4955 generic.go:356] "Generic (PLEG): container finished" podID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerID="a015048f1078b11b579f0442c97a480775308817c7e2bbba49e126c382b663e7" exitCode=0
Nov 03 15:46:05.996804 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.996755    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" event={"ID":"7a27de22-295a-4b0a-bc01-95c05241a6b0","Type":"ContainerDied","Data":"a015048f1078b11b579f0442c97a480775308817c7e2bbba49e126c382b663e7"}
Nov 03 15:46:05.998741 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.998716    4955 generic.go:356] "Generic (PLEG): container finished" podID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerID="f72351ec3d6d4c7237740f50af41ba61b73f1068c0742375e17fffe04456ff3a" exitCode=0
Nov 03 15:46:05.998823 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:05.998770    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" event={"ID":"27f70b0e-88d8-495b-ba4a-72a99a6a6649","Type":"ContainerDied","Data":"f72351ec3d6d4c7237740f50af41ba61b73f1068c0742375e17fffe04456ff3a"}
Nov 03 15:46:06.000695 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.000644    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps\" (UniqueName: \"kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps\") pod \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") "
Nov 03 15:46:06.001083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.001041    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume\") pod \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\" (UID: \"1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb\") "
Nov 03 15:46:06.002312 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.002273    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume" pod "1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" (UID: "1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:06.010378 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.010303    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps" pod "1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" (UID: "1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb"). InnerVolumeSpecName "kube-api-access-hrtps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:06.015251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.011734    4955 scope.go:122] "RemoveContainer" containerID="70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008"
Nov 03 15:46:06.015251 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:06.012721    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008\": container with ID starting with 70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008 not found: ID does not exist" containerID="70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008"
Nov 03 15:46:06.015251 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.012764    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008"} err="failed to get container status \"70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008\": rpc error: code = NotFound desc = could not find container \"70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008\": container with ID starting with 70b0a61b867135fd4955e9cbb2105982ea17e134e914a3cfede5dd2105911008 not found: ID does not exist"
Nov 03 15:46:06.103739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.103697    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtps\" (UniqueName: \"kubernetes.io/projected/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-kube-api-access-hrtps\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.103739 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.103742    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.148042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.139242    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:46:06.206634 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.206595    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs\") pod \"7a27de22-295a-4b0a-bc01-95c05241a6b0\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") "
Nov 03 15:46:06.206814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.206706    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir\") pod \"7a27de22-295a-4b0a-bc01-95c05241a6b0\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") "
Nov 03 15:46:06.206814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.206746    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache\") pod \"7a27de22-295a-4b0a-bc01-95c05241a6b0\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") "
Nov 03 15:46:06.206814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.206781    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home\") pod \"7a27de22-295a-4b0a-bc01-95c05241a6b0\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") "
Nov 03 15:46:06.206814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.206809    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm\") pod \"7a27de22-295a-4b0a-bc01-95c05241a6b0\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") "
Nov 03 15:46:06.207040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.206842    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm\" (UniqueName: \"kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm\") pod \"7a27de22-295a-4b0a-bc01-95c05241a6b0\" (UID: \"7a27de22-295a-4b0a-bc01-95c05241a6b0\") "
Nov 03 15:46:06.208354 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.208061    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home" pod "7a27de22-295a-4b0a-bc01-95c05241a6b0" (UID: "7a27de22-295a-4b0a-bc01-95c05241a6b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.208826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.208790    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir" pod "7a27de22-295a-4b0a-bc01-95c05241a6b0" (UID: "7a27de22-295a-4b0a-bc01-95c05241a6b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.209269 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.209250    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache" pod "7a27de22-295a-4b0a-bc01-95c05241a6b0" (UID: "7a27de22-295a-4b0a-bc01-95c05241a6b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.213565 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.213462    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs" pod "7a27de22-295a-4b0a-bc01-95c05241a6b0" (UID: "7a27de22-295a-4b0a-bc01-95c05241a6b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:06.219185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.219133    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm" pod "7a27de22-295a-4b0a-bc01-95c05241a6b0" (UID: "7a27de22-295a-4b0a-bc01-95c05241a6b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.219671 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.219647    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm" pod "7a27de22-295a-4b0a-bc01-95c05241a6b0" (UID: "7a27de22-295a-4b0a-bc01-95c05241a6b0"). InnerVolumeSpecName "kube-api-access-rzxzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:06.228456 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.228395    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:46:06.233276 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.230656    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:46:06.307952 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.307906    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm\") pod \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") "
Nov 03 15:46:06.308152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.307998    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir\") pod \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") "
Nov 03 15:46:06.308152 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308074    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv\" (UniqueName: \"kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv\") pod \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") "
Nov 03 15:46:06.308277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308167    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir\") pod \"59910387-1111-4c0b-a838-247b1c8173b3\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") "
Nov 03 15:46:06.308277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308200    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache\") pod \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") "
Nov 03 15:46:06.308277 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308228    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm\") pod \"59910387-1111-4c0b-a838-247b1c8173b3\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") "
Nov 03 15:46:06.308442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308318    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home\") pod \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") "
Nov 03 15:46:06.308442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308340    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir" pod "27f70b0e-88d8-495b-ba4a-72a99a6a6649" (UID: "27f70b0e-88d8-495b-ba4a-72a99a6a6649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.308442 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308367    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home\") pod \"59910387-1111-4c0b-a838-247b1c8173b3\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") "
Nov 03 15:46:06.308587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308448    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs\" (UniqueName: \"kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs\") pod \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\" (UID: \"27f70b0e-88d8-495b-ba4a-72a99a6a6649\") "
Nov 03 15:46:06.308587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308481    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache\") pod \"59910387-1111-4c0b-a838-247b1c8173b3\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") "
Nov 03 15:46:06.308587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308515    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k\" (UniqueName: \"kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k\") pod \"59910387-1111-4c0b-a838-247b1c8173b3\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") "
Nov 03 15:46:06.308587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308505    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir" pod "59910387-1111-4c0b-a838-247b1c8173b3" (UID: "59910387-1111-4c0b-a838-247b1c8173b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.308587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.308544    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs\") pod \"59910387-1111-4c0b-a838-247b1c8173b3\" (UID: \"59910387-1111-4c0b-a838-247b1c8173b3\") "
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309217    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309242    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309256    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27de22-295a-4b0a-bc01-95c05241a6b0-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309268    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309280    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309294    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309307    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27de22-295a-4b0a-bc01-95c05241a6b0-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309320    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxzm\" (UniqueName: \"kubernetes.io/projected/7a27de22-295a-4b0a-bc01-95c05241a6b0-kube-api-access-rzxzm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.309875    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home" pod "59910387-1111-4c0b-a838-247b1c8173b3" (UID: "59910387-1111-4c0b-a838-247b1c8173b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.310018    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home" pod "27f70b0e-88d8-495b-ba4a-72a99a6a6649" (UID: "27f70b0e-88d8-495b-ba4a-72a99a6a6649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.310113    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache" pod "59910387-1111-4c0b-a838-247b1c8173b3" (UID: "59910387-1111-4c0b-a838-247b1c8173b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.310656 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.310613    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache" pod "27f70b0e-88d8-495b-ba4a-72a99a6a6649" (UID: "27f70b0e-88d8-495b-ba4a-72a99a6a6649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.311590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.311360    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm" pod "27f70b0e-88d8-495b-ba4a-72a99a6a6649" (UID: "27f70b0e-88d8-495b-ba4a-72a99a6a6649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.311590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.311362    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm" pod "59910387-1111-4c0b-a838-247b1c8173b3" (UID: "59910387-1111-4c0b-a838-247b1c8173b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:06.312208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.312175    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs" pod "27f70b0e-88d8-495b-ba4a-72a99a6a6649" (UID: "27f70b0e-88d8-495b-ba4a-72a99a6a6649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:06.312208 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.312192    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv" pod "27f70b0e-88d8-495b-ba4a-72a99a6a6649" (UID: "27f70b0e-88d8-495b-ba4a-72a99a6a6649"). InnerVolumeSpecName "kube-api-access-49ckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:06.312460 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.312444    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs" pod "59910387-1111-4c0b-a838-247b1c8173b3" (UID: "59910387-1111-4c0b-a838-247b1c8173b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:06.313444 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.313422    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k" pod "59910387-1111-4c0b-a838-247b1c8173b3" (UID: "59910387-1111-4c0b-a838-247b1c8173b3"). InnerVolumeSpecName "kube-api-access-vcc9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410653    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410686    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410696    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410705    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410713    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f70b0e-88d8-495b-ba4a-72a99a6a6649-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410721    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59910387-1111-4c0b-a838-247b1c8173b3-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410730    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcc9k\" (UniqueName: \"kubernetes.io/projected/59910387-1111-4c0b-a838-247b1c8173b3-kube-api-access-vcc9k\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.410735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410739    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59910387-1111-4c0b-a838-247b1c8173b3-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.411123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410747    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f70b0e-88d8-495b-ba4a-72a99a6a6649-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.411123 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.410757    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ckv\" (UniqueName: \"kubernetes.io/projected/27f70b0e-88d8-495b-ba4a-72a99a6a6649-kube-api-access-49ckv\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:06.431761 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.431365    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"]
Nov 03 15:46:06.465219 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:06.465148    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-78034f-ffc8cd7d7-n26vk"]
Nov 03 15:46:07.005304 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.005275    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"
Nov 03 15:46:07.005796 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.005313    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" event={"ID":"7a27de22-295a-4b0a-bc01-95c05241a6b0","Type":"ContainerDied","Data":"2a05b0c18d4ff83bda235d337194f19b16619438b7878eed5cd15d9377c41092"}
Nov 03 15:46:07.005796 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.005360    4955 scope.go:122] "RemoveContainer" containerID="a015048f1078b11b579f0442c97a480775308817c7e2bbba49e126c382b663e7"
Nov 03 15:46:07.007168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.007145    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"
Nov 03 15:46:07.007168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.007153    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" event={"ID":"27f70b0e-88d8-495b-ba4a-72a99a6a6649","Type":"ContainerDied","Data":"47c08b384e5f1cf2b247fd41f49a1f575c370d64e5c69d43e9759732a376bf45"}
Nov 03 15:46:07.008850 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.008828    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"
Nov 03 15:46:07.008963 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.008828    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695" event={"ID":"59910387-1111-4c0b-a838-247b1c8173b3","Type":"ContainerDied","Data":"195f0db547b5f51a2f3cd6e39c4c0b09695d200ea59b3c3d19edb3e5022a6c0b"}
Nov 03 15:46:07.018540 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.018516    4955 scope.go:122] "RemoveContainer" containerID="f72351ec3d6d4c7237740f50af41ba61b73f1068c0742375e17fffe04456ff3a"
Nov 03 15:46:07.028077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.028054    4955 scope.go:122] "RemoveContainer" containerID="7c6139d0d5969877638bee9eb0138033eaaac059bcc5bc2ec1bc9eaf4c11debc"
Nov 03 15:46:07.098568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.098528    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 15:46:07.107384 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.107343    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:46:07.174621 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.174576    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"]
Nov 03 15:46:07.181172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.181137    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-w1-9855ff-kserve-fd8457759-zjd8h"]
Nov 03 15:46:07.206425 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.206393    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" path="/var/lib/kubelet/pods/1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb/volumes"
Nov 03 15:46:07.206875 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.206860    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" path="/var/lib/kubelet/pods/27f70b0e-88d8-495b-ba4a-72a99a6a6649/volumes"
Nov 03 15:46:07.292781 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.292693    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"]
Nov 03 15:46:07.297592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.297563    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/premium-w1-9855ff-kserve-78f5bdd44d-mh695"]
Nov 03 15:46:07.408027 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.407991    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"]
Nov 03 15:46:07.413009 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:07.412983    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9855ff/distinct-2-w1-9855ff-kserve-76b46d699b-ngz2m"]
Nov 03 15:46:09.205223 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:09.205191    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59910387-1111-4c0b-a838-247b1c8173b3" path="/var/lib/kubelet/pods/59910387-1111-4c0b-a838-247b1c8173b3/volumes"
Nov 03 15:46:09.205648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:09.205624    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" path="/var/lib/kubelet/pods/7a27de22-295a-4b0a-bc01-95c05241a6b0/volumes"
Nov 03 15:46:15.681524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:15.681439    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"]
Nov 03 15:46:15.681983 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:15.681701    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" containerID="cri-o://e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede" gracePeriod=30
Nov 03 15:46:16.969313 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:16.969248    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main" containerID="cri-o://2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96" gracePeriod=45
Nov 03 15:46:17.009708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.009667    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main" probeResult="failure" output="Get \"https://10.132.0.117:8000/ready\": dial tcp 10.132.0.117:8000: connect: connection refused"
Nov 03 15:46:17.049858 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.049824    4955 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:46:17.050034 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.049881    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:46:17.239680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.239654    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:46:17.390765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.390725    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache\") pod \"82e67cca-1021-487e-9c5a-8789af0a3426\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") "
Nov 03 15:46:17.390998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.390889    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir\") pod \"82e67cca-1021-487e-9c5a-8789af0a3426\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") "
Nov 03 15:46:17.390998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.390945    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home\") pod \"82e67cca-1021-487e-9c5a-8789af0a3426\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") "
Nov 03 15:46:17.391084 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391052    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm\") pod \"82e67cca-1021-487e-9c5a-8789af0a3426\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") "
Nov 03 15:46:17.391132 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391116    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555\" (UniqueName: \"kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555\") pod \"82e67cca-1021-487e-9c5a-8789af0a3426\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") "
Nov 03 15:46:17.391170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391154    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs\" (UniqueName: \"kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs\") pod \"82e67cca-1021-487e-9c5a-8789af0a3426\" (UID: \"82e67cca-1021-487e-9c5a-8789af0a3426\") "
Nov 03 15:46:17.391393 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391373    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home" pod "82e67cca-1021-487e-9c5a-8789af0a3426" (UID: "82e67cca-1021-487e-9c5a-8789af0a3426"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:17.391674 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391654    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:17.391893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391868    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache" pod "82e67cca-1021-487e-9c5a-8789af0a3426" (UID: "82e67cca-1021-487e-9c5a-8789af0a3426"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:17.391893 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.391880    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir" pod "82e67cca-1021-487e-9c5a-8789af0a3426" (UID: "82e67cca-1021-487e-9c5a-8789af0a3426"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:17.398058 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.398021    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm" pod "82e67cca-1021-487e-9c5a-8789af0a3426" (UID: "82e67cca-1021-487e-9c5a-8789af0a3426"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:17.398227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.398078    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs" pod "82e67cca-1021-487e-9c5a-8789af0a3426" (UID: "82e67cca-1021-487e-9c5a-8789af0a3426"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:17.398227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.398210    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555" pod "82e67cca-1021-487e-9c5a-8789af0a3426" (UID: "82e67cca-1021-487e-9c5a-8789af0a3426"). InnerVolumeSpecName "kube-api-access-2c555". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:17.493089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.493011    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:17.493089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.493048    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c555\" (UniqueName: \"kubernetes.io/projected/82e67cca-1021-487e-9c5a-8789af0a3426-kube-api-access-2c555\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:17.493089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.493066    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e67cca-1021-487e-9c5a-8789af0a3426-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:17.493089 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.493081    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:17.493332 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:17.493094    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e67cca-1021-487e-9c5a-8789af0a3426-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:18.089681 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.089640    4955 generic.go:356] "Generic (PLEG): container finished" podID="82e67cca-1021-487e-9c5a-8789af0a3426" containerID="2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96" exitCode=0
Nov 03 15:46:18.090180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.089692    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"
Nov 03 15:46:18.090180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.089720    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" event={"ID":"82e67cca-1021-487e-9c5a-8789af0a3426","Type":"ContainerDied","Data":"2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96"}
Nov 03 15:46:18.090180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.089765    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh" event={"ID":"82e67cca-1021-487e-9c5a-8789af0a3426","Type":"ContainerDied","Data":"4622410b89361ae8cb29170f1d689820f6fdfec8b32c30f5de22402683e341c6"}
Nov 03 15:46:18.090180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.089782    4955 scope.go:122] "RemoveContainer" containerID="2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96"
Nov 03 15:46:18.100475 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.100449    4955 scope.go:122] "RemoveContainer" containerID="2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96"
Nov 03 15:46:18.100778 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:18.100754    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96\": container with ID starting with 2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96 not found: ID does not exist" containerID="2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96"
Nov 03 15:46:18.100852 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.100790    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96"} err="failed to get container status \"2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96\": rpc error: code = NotFound desc = could not find container \"2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96\": container with ID starting with 2e4627c320fcba8b240ead45c51c5d05a0985873f1855804adb14146846a8f96 not found: ID does not exist"
Nov 03 15:46:18.282735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.282688    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"]
Nov 03 15:46:18.332860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:18.332823    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5cad072f/e2e-lifecycle-model-5cad072f-kserve-7b6b87c677-f2zzh"]
Nov 03 15:46:19.205182 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:19.205151    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" path="/var/lib/kubelet/pods/82e67cca-1021-487e-9c5a-8789af0a3426/volumes"
Nov 03 15:46:20.959292 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:20.959234    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:46:21.109271 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.109241    4955 generic.go:356] "Generic (PLEG): container finished" podID="b578c022-7306-43d9-8886-4f16c09db6c1" containerID="e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede" exitCode=0
Nov 03 15:46:21.109443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.109298    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"
Nov 03 15:46:21.109443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.109322    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" event={"ID":"b578c022-7306-43d9-8886-4f16c09db6c1","Type":"ContainerDied","Data":"e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede"}
Nov 03 15:46:21.109443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.109377    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7" event={"ID":"b578c022-7306-43d9-8886-4f16c09db6c1","Type":"ContainerDied","Data":"64b303cf2c02f5788d91d4edce6326489899fc77e4fcded1f113f358e1f3cc36"}
Nov 03 15:46:21.109443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.109403    4955 scope.go:122] "RemoveContainer" containerID="e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede"
Nov 03 15:46:21.119737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.119710    4955 scope.go:122] "RemoveContainer" containerID="4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"
Nov 03 15:46:21.129499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.129471    4955 scope.go:122] "RemoveContainer" containerID="e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede"
Nov 03 15:46:21.129887 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:21.129863    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede\": container with ID starting with e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede not found: ID does not exist" containerID="e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede"
Nov 03 15:46:21.130004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.129901    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede"} err="failed to get container status \"e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede\": rpc error: code = NotFound desc = could not find container \"e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede\": container with ID starting with e956f26bd7a8263e7c59424367236b58c4f1b4ea43a1b2853f783c07fbe59ede not found: ID does not exist"
Nov 03 15:46:21.130004 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.129922    4955 scope.go:122] "RemoveContainer" containerID="4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"
Nov 03 15:46:21.130300 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:21.130280    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86\": container with ID starting with 4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86 not found: ID does not exist" containerID="4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"
Nov 03 15:46:21.130352 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.130307    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86"} err="failed to get container status \"4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86\": rpc error: code = NotFound desc = could not find container \"4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86\": container with ID starting with 4ea6d2126c3a8ddb9a3330dc3227d890d0435d0a2813503e4224323c8b2ccb86 not found: ID does not exist"
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140420    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140477    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140629    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140684    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140751    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140795    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140825    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140865    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.140897    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket\") pod \"b578c022-7306-43d9-8886-4f16c09db6c1\" (UID: \"b578c022-7306-43d9-8886-4f16c09db6c1\") "
Nov 03 15:46:21.142000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.141644    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:21.143890 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.143675    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.144302 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.144268    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:21.144625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.144599    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:21.144836 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.144791    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:21.145111 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.145090    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:21.151952 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.151922    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:21.152101 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.152076    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "kube-api-access-mtd2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:21.152455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.152429    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:46:21.152573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.152549    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy" pod "b578c022-7306-43d9-8886-4f16c09db6c1" (UID: "b578c022-7306-43d9-8886-4f16c09db6c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:21.245426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245394    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245425    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtd2b\" (UniqueName: \"kubernetes.io/projected/b578c022-7306-43d9-8886-4f16c09db6c1-kube-api-access-mtd2b\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245436    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b578c022-7306-43d9-8886-4f16c09db6c1-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245445    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245454    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245463    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b578c022-7306-43d9-8886-4f16c09db6c1-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245473    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.245625 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.245480    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b578c022-7306-43d9-8886-4f16c09db6c1-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:21.529421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.529339    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"]
Nov 03 15:46:21.535257 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:21.535230    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f356bb-openshift-default-cb4bc9f6f-h5ln7"]
Nov 03 15:46:23.205401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:23.205356    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" path="/var/lib/kubelet/pods/b578c022-7306-43d9-8886-4f16c09db6c1/volumes"
Nov 03 15:46:23.688506 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:23.688469    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"]
Nov 03 15:46:23.688726 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:23.688705    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" podUID="bd18e2cb-ed33-4461-b18f-1458b96322ed" containerName="maas-api" containerID="cri-o://bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f" gracePeriod=30
Nov 03 15:46:23.983188 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:23.983161    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:46:24.094998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.083393    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs\") pod \"bd18e2cb-ed33-4461-b18f-1458b96322ed\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") "
Nov 03 15:46:24.094998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.083456    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97\" (UniqueName: \"kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97\") pod \"bd18e2cb-ed33-4461-b18f-1458b96322ed\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") "
Nov 03 15:46:24.094998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.083529    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca\") pod \"bd18e2cb-ed33-4461-b18f-1458b96322ed\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") "
Nov 03 15:46:24.094998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.083660    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\") pod \"bd18e2cb-ed33-4461-b18f-1458b96322ed\" (UID: \"bd18e2cb-ed33-4461-b18f-1458b96322ed\") "
Nov 03 15:46:24.095318 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.095085    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca" pod "bd18e2cb-ed33-4461-b18f-1458b96322ed" (UID: "bd18e2cb-ed33-4461-b18f-1458b96322ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:24.113998 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.106532    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls" pod "bd18e2cb-ed33-4461-b18f-1458b96322ed" (UID: "bd18e2cb-ed33-4461-b18f-1458b96322ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:24.119430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.119366    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs" pod "bd18e2cb-ed33-4461-b18f-1458b96322ed" (UID: "bd18e2cb-ed33-4461-b18f-1458b96322ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:24.159038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.158985    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97" pod "bd18e2cb-ed33-4461-b18f-1458b96322ed" (UID: "bd18e2cb-ed33-4461-b18f-1458b96322ed"). InnerVolumeSpecName "kube-api-access-6xb97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:24.166545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.166085    4955 generic.go:356] "Generic (PLEG): container finished" podID="bd18e2cb-ed33-4461-b18f-1458b96322ed" containerID="bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f" exitCode=0
Nov 03 15:46:24.166545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.166248    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"
Nov 03 15:46:24.166545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.166293    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" event={"ID":"bd18e2cb-ed33-4461-b18f-1458b96322ed","Type":"ContainerDied","Data":"bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f"}
Nov 03 15:46:24.166545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.166342    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28" event={"ID":"bd18e2cb-ed33-4461-b18f-1458b96322ed","Type":"ContainerDied","Data":"c403f3affddbad93d5dce8b872b586285640a6066f16e8b4705755c3ee4ca25b"}
Nov 03 15:46:24.166545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.166374    4955 scope.go:122] "RemoveContainer" containerID="bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f"
Nov 03 15:46:24.183743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.183709    4955 scope.go:122] "RemoveContainer" containerID="bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f"
Nov 03 15:46:24.184407 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:24.184368    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f\": container with ID starting with bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f not found: ID does not exist" containerID="bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f"
Nov 03 15:46:24.184533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.184420    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f"} err="failed to get container status \"bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f\": rpc error: code = NotFound desc = could not find container \"bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f\": container with ID starting with bd7a88ba111d9f1ff66b9e734fbb20ee1e81f4d8a679b6aa2464753e1a63d13f not found: ID does not exist"
Nov 03 15:46:24.184808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.184788    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xb97\" (UniqueName: \"kubernetes.io/projected/bd18e2cb-ed33-4461-b18f-1458b96322ed-kube-api-access-6xb97\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:24.184865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.184812    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd18e2cb-ed33-4461-b18f-1458b96322ed-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:24.184865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.184823    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:24.184865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.184832    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd18e2cb-ed33-4461-b18f-1458b96322ed-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:24.346567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.346479    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"]
Nov 03 15:46:24.361142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.361110    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-78034f-76f5fb6d6f-zps28"]
Nov 03 15:46:24.560838 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.560791    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"]
Nov 03 15:46:24.562545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562516    4955 cpu_manager.go:414] "containerMap: removing container" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:24.562545 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562542    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562556    4955 cpu_manager.go:414] "containerMap: removing container" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562563    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562573    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562581    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562600    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562609    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562633    4955 cpu_manager.go:414] "containerMap: removing container" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562642    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562652    4955 cpu_manager.go:414] "containerMap: removing container" podUID="bd18e2cb-ed33-4461-b18f-1458b96322ed" containerName="maas-api"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562658    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd18e2cb-ed33-4461-b18f-1458b96322ed" containerName="maas-api"
Nov 03 15:46:24.562757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562678    4955 cpu_manager.go:414] "containerMap: removing container" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562845    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562869    4955 cpu_manager.go:414] "containerMap: removing container" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562877    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562893    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562901    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562929    4955 cpu_manager.go:414] "containerMap: removing container" podUID="1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" containerName="payload-processing"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562937    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" containerName="payload-processing"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562956    4955 cpu_manager.go:414] "containerMap: removing container" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.562983    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563007    4955 cpu_manager.go:414] "containerMap: removing container" podUID="d95bec2b-299d-4046-bbdb-96d9783f8880" containerName="maas-api"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563015    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95bec2b-299d-4046-bbdb-96d9783f8880" containerName="maas-api"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563027    4955 cpu_manager.go:414] "containerMap: removing container" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563036    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563102    4955 cpu_manager.go:414] "containerMap: removing container" podUID="8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" containerName="maas-api"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563110    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" containerName="maas-api"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563128    4955 cpu_manager.go:414] "containerMap: removing container" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563136    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563187    4955 cpu_manager.go:414] "containerMap: removing container" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563201    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563229    4955 cpu_manager.go:414] "containerMap: removing container" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563237    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563260    4955 cpu_manager.go:414] "containerMap: removing container" podUID="3c8117ac-048b-48fb-9316-90f39450be94" containerName="maas-api"
Nov 03 15:46:24.563337 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563270    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8117ac-048b-48fb-9316-90f39450be94" containerName="maas-api"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563815    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563836    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e67cca-1021-487e-9c5a-8789af0a3426" containerName="main"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563856    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563876    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563922    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563934    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd18e2cb-ed33-4461-b18f-1458b96322ed" containerName="maas-api"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563944    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.563959    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564051    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a27de22-295a-4b0a-bc01-95c05241a6b0" containerName="main"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564068    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7a88c1-66e3-44a7-acf3-e1d3f2e315eb" containerName="payload-processing"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564086    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564098    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95bec2b-299d-4046-bbdb-96d9783f8880" containerName="maas-api"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564107    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="59910387-1111-4c0b-a838-247b1c8173b3" containerName="main"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564136    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f46fe3a-c661-4ce1-8a74-f80c501c3c2e" containerName="maas-api"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564146    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f70b0e-88d8-495b-ba4a-72a99a6a6649" containerName="main"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564166    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564185    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="f909bbbf-ca76-48ac-9e32-4aa4d5613c67" containerName="istio-proxy"
Nov 03 15:46:24.564524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.564196    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8117ac-048b-48fb-9316-90f39450be94" containerName="maas-api"
Nov 03 15:46:24.570834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.570803    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.588729 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.588701    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.588840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.588741    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.588840 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.588819    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.588930 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.588856    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.588989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.588922    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.589042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.589028    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78zf\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.589097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.589087    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.589175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.589162    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.589215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.589202    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.625627 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.625543    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9855ff-openshift-default-dockercfg-5hjbk\""
Nov 03 15:46:24.628989 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.626167    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"]
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.708753    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.708820    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.708852    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.708906    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.709009    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78zf\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.709099    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.709126    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.709182    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.709993 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.709219    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.711496 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.711369    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.712794 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.712721    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.713220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.713196    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.713590 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.713569    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.717299 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.717277    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.717685 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.717660    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.728371 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.728343    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.783450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.781934    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.783450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.782358    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78zf\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf\") pod \"e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") " pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:24.887431 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:24.887337    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:25.205167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:25.205132    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd18e2cb-ed33-4461-b18f-1458b96322ed" path="/var/lib/kubelet/pods/bd18e2cb-ed33-4461-b18f-1458b96322ed/volumes"
Nov 03 15:46:25.321260 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:25.320665    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"]
Nov 03 15:46:25.324108 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:46:25.324078    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49086954_ec28_4a57_8c3c_2c3fb185d67d.slice/crio-b9595708facb33d9631b788ea8fb17016630cad72cfd72a323d9a520063fc828 WatchSource:0}: Error finding container b9595708facb33d9631b788ea8fb17016630cad72cfd72a323d9a520063fc828: Status 404 returned error can't find the container with id b9595708facb33d9631b788ea8fb17016630cad72cfd72a323d9a520063fc828
Nov 03 15:46:25.326802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:25.326757    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:46:25.326922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:25.326895    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:46:25.327015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:25.326994    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:46:26.180358 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:26.180323    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" event={"ID":"49086954-ec28-4a57-8c3c-2c3fb185d67d","Type":"ContainerStarted","Data":"b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d"}
Nov 03 15:46:26.180752 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:26.180379    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" event={"ID":"49086954-ec28-4a57-8c3c-2c3fb185d67d","Type":"ContainerStarted","Data":"b9595708facb33d9631b788ea8fb17016630cad72cfd72a323d9a520063fc828"}
Nov 03 15:46:26.419798 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:26.419735    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podStartSLOduration=2.419716015 podStartE2EDuration="2.419716015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:46:26.367944032 +0000 UTC m=+2301.292761945" watchObservedRunningTime="2026-11-03 15:46:26.419716015 +0000 UTC m=+2301.344533930"
Nov 03 15:46:26.887924 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:26.887887    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:26.889604 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:26.889574    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 15:46:26.889743 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:26.889622    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 15:46:27.887863 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:27.887829    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 15:46:27.888320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:27.887883    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 15:46:27.964387 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:27.964345    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:46:28.904036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:28.902326    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 15:46:28.904036 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:28.902383    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 15:46:29.887742 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:29.887702    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 15:46:29.887962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:29.887780    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 15:46:30.662043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:30.661992    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"]
Nov 03 15:46:30.662499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:30.662259    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" podUID="4a6a00f6-f94e-4663-8398-cb49f8d080c6" containerName="maas-api" containerID="cri-o://2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea" gracePeriod=30
Nov 03 15:46:30.890928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:30.889395    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 15:46:30.890928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:30.889473    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 15:46:31.038001 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.036963    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:46:31.102516 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.102482    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv\" (UniqueName: \"kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv\") pod \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") "
Nov 03 15:46:31.103227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.103202    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca\") pod \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") "
Nov 03 15:46:31.105881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.103418    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs\") pod \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") "
Nov 03 15:46:31.106147 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.105314    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca" pod "4a6a00f6-f94e-4663-8398-cb49f8d080c6" (UID: "4a6a00f6-f94e-4663-8398-cb49f8d080c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:31.106574 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.106554    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\") pod \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\" (UID: \"4a6a00f6-f94e-4663-8398-cb49f8d080c6\") "
Nov 03 15:46:31.107374 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.107350    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6a00f6-f94e-4663-8398-cb49f8d080c6-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:31.110687 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.110646    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls" pod "4a6a00f6-f94e-4663-8398-cb49f8d080c6" (UID: "4a6a00f6-f94e-4663-8398-cb49f8d080c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:31.112024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.111806    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv" pod "4a6a00f6-f94e-4663-8398-cb49f8d080c6" (UID: "4a6a00f6-f94e-4663-8398-cb49f8d080c6"). InnerVolumeSpecName "kube-api-access-5k7lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:31.115108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.115081    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs" pod "4a6a00f6-f94e-4663-8398-cb49f8d080c6" (UID: "4a6a00f6-f94e-4663-8398-cb49f8d080c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:31.208907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.208854    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7lv\" (UniqueName: \"kubernetes.io/projected/4a6a00f6-f94e-4663-8398-cb49f8d080c6-kube-api-access-5k7lv\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:31.208907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.208890    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:31.208907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.208901    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6a00f6-f94e-4663-8398-cb49f8d080c6-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:31.360403 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.360299    4955 generic.go:356] "Generic (PLEG): container finished" podID="4a6a00f6-f94e-4663-8398-cb49f8d080c6" containerID="2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea" exitCode=0
Nov 03 15:46:31.360403 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.360346    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"
Nov 03 15:46:31.360403 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.360376    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" event={"ID":"4a6a00f6-f94e-4663-8398-cb49f8d080c6","Type":"ContainerDied","Data":"2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea"}
Nov 03 15:46:31.360653 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.360430    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd" event={"ID":"4a6a00f6-f94e-4663-8398-cb49f8d080c6","Type":"ContainerDied","Data":"19dc483f8a7382040209eaa3fa42ebfacd68a4deda1c63ecdbee6839d478de67"}
Nov 03 15:46:31.360653 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.360451    4955 scope.go:122] "RemoveContainer" containerID="2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea"
Nov 03 15:46:31.369821 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.369801    4955 scope.go:122] "RemoveContainer" containerID="2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea"
Nov 03 15:46:31.370144 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:31.370125    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea\": container with ID starting with 2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea not found: ID does not exist" containerID="2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea"
Nov 03 15:46:31.370185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.370157    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea"} err="failed to get container status \"2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea\": rpc error: code = NotFound desc = could not find container \"2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea\": container with ID starting with 2da368c4f1006f5245361acdceaf20be90fb5f293207d27beaa20898dc1c42ea not found: ID does not exist"
Nov 03 15:46:31.556617 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.556554    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"]
Nov 03 15:46:31.558837 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.558810    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90fcca-fb7f894d8-x2xrd"]
Nov 03 15:46:31.888323 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.888290    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 15:46:31.888806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:31.888353    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 15:46:33.098848 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:33.098811    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:33.099693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:33.099668    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:33.100416 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:33.100395    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:46:33.205533 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:33.205504    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6a00f6-f94e-4663-8398-cb49f8d080c6" path="/var/lib/kubelet/pods/4a6a00f6-f94e-4663-8398-cb49f8d080c6/volumes"
Nov 03 15:46:42.716605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:42.716559    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"]
Nov 03 15:46:42.717163 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:42.716918    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy" containerID="cri-o://bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9" gracePeriod=30
Nov 03 15:46:42.723779 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:42.723747    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"]
Nov 03 15:46:42.724091 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:42.724040    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy" containerID="cri-o://a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3" gracePeriod=30
Nov 03 15:46:42.733042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:42.733015    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"]
Nov 03 15:46:42.733231 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:42.733198    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy" containerID="cri-o://fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808" gracePeriod=30
Nov 03 15:46:43.259497 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:43.259452    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"]
Nov 03 15:46:48.006558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.006525    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:46:48.051357 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.051337    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:46:48.058080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.058061    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:46:48.156367 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156336    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.156568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156421    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.156568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156458    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.156568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156485    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.156568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156515    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.156568 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156552    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156578    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156612    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156647    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156678    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156698    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156710    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156747    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156773    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156764    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156791    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156808    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.156866 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156824    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156873    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156922    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.156957    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157012    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157047    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157050    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157120    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157186    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157304    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157334    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157369    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token\") pod \"4734a279-01a0-4d25-a81d-3d15cc9634da\" (UID: \"4734a279-01a0-4d25-a81d-3d15cc9634da\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157372    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157402    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy\") pod \"425b176f-1f9d-449c-8340-9a862828ca56\" (UID: \"425b176f-1f9d-449c-8340-9a862828ca56\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157456    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.157519 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157505    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo\") pod \"caab1123-8037-409b-af2e-31de277de6c8\" (UID: \"caab1123-8037-409b-af2e-31de277de6c8\") "
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.157621    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158082    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158042    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158103    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158136    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158154    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158170    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158185    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158200    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4734a279-01a0-4d25-a81d-3d15cc9634da-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.158505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158292    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.159052 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.158952    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.160684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.159111    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:48.160684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.159394    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.160684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.159943    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.160684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.160009    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.160684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.160597    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.161520 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.161489    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.162226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162206    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.162410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162233    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:48.162477 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162415    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:46:48.162532 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162484    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:46:48.163124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162598    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "kube-api-access-hznrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:48.163124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162824    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "kube-api-access-9rvlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:48.163124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162850    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:48.163124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.162914    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token" pod "4734a279-01a0-4d25-a81d-3d15cc9634da" (UID: "4734a279-01a0-4d25-a81d-3d15cc9634da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:48.163455 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.163434    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:48.163633 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.163610    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq" pod "425b176f-1f9d-449c-8340-9a862828ca56" (UID: "425b176f-1f9d-449c-8340-9a862828ca56"). InnerVolumeSpecName "kube-api-access-k58lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:48.164401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.164385    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo" pod "caab1123-8037-409b-af2e-31de277de6c8" (UID: "caab1123-8037-409b-af2e-31de277de6c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258874    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258907    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258923    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k58lq\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-kube-api-access-k58lq\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258938    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258950    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258964    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caab1123-8037-409b-af2e-31de277de6c8-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.258992    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caab1123-8037-409b-af2e-31de277de6c8-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259006    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259019    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259032    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259047    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvlp\" (UniqueName: \"kubernetes.io/projected/4734a279-01a0-4d25-a81d-3d15cc9634da-kube-api-access-9rvlp\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259060    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259072    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259084    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4734a279-01a0-4d25-a81d-3d15cc9634da-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259099    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caab1123-8037-409b-af2e-31de277de6c8-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259112    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznrb\" (UniqueName: \"kubernetes.io/projected/caab1123-8037-409b-af2e-31de277de6c8-kube-api-access-hznrb\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259129    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425b176f-1f9d-449c-8340-9a862828ca56-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259143    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425b176f-1f9d-449c-8340-9a862828ca56-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259154    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425b176f-1f9d-449c-8340-9a862828ca56-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.259226 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.259162    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425b176f-1f9d-449c-8340-9a862828ca56-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:48.277721 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.277687    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"]
Nov 03 15:46:48.462703 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.462670    4955 generic.go:356] "Generic (PLEG): container finished" podID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerID="a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3" exitCode=0
Nov 03 15:46:48.462892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.462730    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"
Nov 03 15:46:48.462892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.462752    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" event={"ID":"4734a279-01a0-4d25-a81d-3d15cc9634da","Type":"ContainerDied","Data":"a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3"}
Nov 03 15:46:48.462892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.462809    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x" event={"ID":"4734a279-01a0-4d25-a81d-3d15cc9634da","Type":"ContainerDied","Data":"267a919b3001e89a51aeddf4c16f80fb945d8e7c40b365d2c7d3b292a8a2a9da"}
Nov 03 15:46:48.462892 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.462838    4955 scope.go:122] "RemoveContainer" containerID="a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3"
Nov 03 15:46:48.469628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.469598    4955 generic.go:356] "Generic (PLEG): container finished" podID="425b176f-1f9d-449c-8340-9a862828ca56" containerID="fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808" exitCode=0
Nov 03 15:46:48.469806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.469789    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"
Nov 03 15:46:48.470426 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.470388    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" event={"ID":"425b176f-1f9d-449c-8340-9a862828ca56","Type":"ContainerDied","Data":"fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808"}
Nov 03 15:46:48.470539 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.470463    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc" event={"ID":"425b176f-1f9d-449c-8340-9a862828ca56","Type":"ContainerDied","Data":"7ffc257d3d101382ff0bb70580be9518ead6b35cda75f5e2b69c7cb0aae66653"}
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.481863    4955 generic.go:356] "Generic (PLEG): container finished" podID="caab1123-8037-409b-af2e-31de277de6c8" containerID="bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9" exitCode=0
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.481909    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" event={"ID":"caab1123-8037-409b-af2e-31de277de6c8","Type":"ContainerDied","Data":"bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9"}
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.481956    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g" event={"ID":"caab1123-8037-409b-af2e-31de277de6c8","Type":"ContainerDied","Data":"e7902a72d426069cf949f962bb5556112c56bd160fc9c9b3e3cb59496ccbff6e"}
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.482051    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.482929    4955 scope.go:122] "RemoveContainer" containerID="a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3"
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:48.483279    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3\": container with ID starting with a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3 not found: ID does not exist" containerID="a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3"
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.483325    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3"} err="failed to get container status \"a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3\": rpc error: code = NotFound desc = could not find container \"a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3\": container with ID starting with a5879cc01fdc25f42e1be87d5b44459af9c8bab6c980665e963fe189ee6335e3 not found: ID does not exist"
Nov 03 15:46:48.487033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.483364    4955 scope.go:122] "RemoveContainer" containerID="fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808"
Nov 03 15:46:48.494395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.494374    4955 scope.go:122] "RemoveContainer" containerID="fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808"
Nov 03 15:46:48.494695 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:48.494673    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808\": container with ID starting with fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808 not found: ID does not exist" containerID="fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808"
Nov 03 15:46:48.494783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.494701    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808"} err="failed to get container status \"fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808\": rpc error: code = NotFound desc = could not find container \"fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808\": container with ID starting with fa4acd277628c246cced406c62f4c4dd928431aebc2f502427d766f082803808 not found: ID does not exist"
Nov 03 15:46:48.494783 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.494721    4955 scope.go:122] "RemoveContainer" containerID="bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9"
Nov 03 15:46:48.506175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.506156    4955 scope.go:122] "RemoveContainer" containerID="592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"
Nov 03 15:46:48.516876 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.516849    4955 scope.go:122] "RemoveContainer" containerID="bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9"
Nov 03 15:46:48.517197 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:48.517178    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9\": container with ID starting with bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9 not found: ID does not exist" containerID="bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9"
Nov 03 15:46:48.517266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.517206    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9"} err="failed to get container status \"bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9\": rpc error: code = NotFound desc = could not find container \"bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9\": container with ID starting with bea6e11db52ac13d0afe41ff5bf646a919777877f9264d422617dcfb7d2fc6c9 not found: ID does not exist"
Nov 03 15:46:48.517266 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.517231    4955 scope.go:122] "RemoveContainer" containerID="592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"
Nov 03 15:46:48.517483 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:48.517467    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee\": container with ID starting with 592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee not found: ID does not exist" containerID="592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"
Nov 03 15:46:48.517535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.517488    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee"} err="failed to get container status \"592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee\": rpc error: code = NotFound desc = could not find container \"592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee\": container with ID starting with 592b3d20d44ee610e2502a7f284ddfbe3a401e348a94cd04dc9ea53a9cbc43ee not found: ID does not exist"
Nov 03 15:46:48.837560 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.837474    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"]
Nov 03 15:46:48.864073 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.863744    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-rt48g"]
Nov 03 15:46:48.968327 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.968206    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"]
Nov 03 15:46:48.970523 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:48.970493    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-642nc"]
Nov 03 15:46:49.124298 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:49.124215    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"]
Nov 03 15:46:49.145192 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:49.145157    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90fcca-openshift-default-575d955f5-2hx2x"]
Nov 03 15:46:49.205307 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:49.205273    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425b176f-1f9d-449c-8340-9a862828ca56" path="/var/lib/kubelet/pods/425b176f-1f9d-449c-8340-9a862828ca56/volumes"
Nov 03 15:46:49.205766 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:49.205752    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" path="/var/lib/kubelet/pods/4734a279-01a0-4d25-a81d-3d15cc9634da/volumes"
Nov 03 15:46:49.206238 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:49.206223    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caab1123-8037-409b-af2e-31de277de6c8" path="/var/lib/kubelet/pods/caab1123-8037-409b-af2e-31de277de6c8/volumes"
Nov 03 15:46:50.777440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777407    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"]
Nov 03 15:46:50.777920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777887    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy"
Nov 03 15:46:50.777920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777898    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy"
Nov 03 15:46:50.777920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777918    4955 cpu_manager.go:414] "containerMap: removing container" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy"
Nov 03 15:46:50.777920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777922    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777937    4955 cpu_manager.go:414] "containerMap: removing container" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777942    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777947    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777952    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777960    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777964    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777990    4955 cpu_manager.go:414] "containerMap: removing container" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.777994    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168c6154-e58d-4ca2-86e3-4893f1b7d9ae" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778002    4955 cpu_manager.go:414] "containerMap: removing container" podUID="4a6a00f6-f94e-4663-8398-cb49f8d080c6" containerName="maas-api"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778006    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6a00f6-f94e-4663-8398-cb49f8d080c6" containerName="maas-api"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778013    4955 cpu_manager.go:414] "containerMap: removing container" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778017    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778029    4955 cpu_manager.go:414] "containerMap: removing container" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778034    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778099    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy"
Nov 03 15:46:50.778102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778107    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b578c022-7306-43d9-8886-4f16c09db6c1" containerName="istio-proxy"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778113    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba150f8-5538-4677-90c9-178795e8c449" containerName="istio-proxy"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778119    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="247f7ba1-a6ff-4d7d-9411-8f1c7311777e" containerName="cleanup"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778125    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6a00f6-f94e-4663-8398-cb49f8d080c6" containerName="maas-api"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778130    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="caab1123-8037-409b-af2e-31de277de6c8" containerName="istio-proxy"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778136    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa52f94-f41d-46fa-8b35-6b72cdbc05c1" containerName="istio-proxy"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778143    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="425b176f-1f9d-449c-8340-9a862828ca56" containerName="istio-proxy"
Nov 03 15:46:50.778587 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.778150    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="4734a279-01a0-4d25-a81d-3d15cc9634da" containerName="istio-proxy"
Nov 03 15:46:50.782951 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.782930    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.837053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.837024    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:46:50.886765 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.886724    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pbkk9\""
Nov 03 15:46:50.888529 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.888505    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dxn\" (UniqueName: \"kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.888688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.888565    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.888751 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.888733    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.889194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.889173    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"]
Nov 03 15:46:50.989843 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.989806    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.990043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.989891    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dxn\" (UniqueName: \"kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.990043 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.989926    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.990635 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.990608    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:50.992443 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:50.992420    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:51.055778 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:51.055707    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dxn\" (UniqueName: \"kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn\") pod \"maas-api-revoke-sub-b9d29c20-46p7f\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:51.096391 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:51.096354    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:51.521129 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:46:51.521082    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19731d76_6fcb_4e10_af2a_39e3c0082b33.slice/crio-0301d75759be5e328b844c1bd8aa8365b50b43782c0ea15dab29f26247526231 WatchSource:0}: Error finding container 0301d75759be5e328b844c1bd8aa8365b50b43782c0ea15dab29f26247526231: Status 404 returned error can't find the container with id 0301d75759be5e328b844c1bd8aa8365b50b43782c0ea15dab29f26247526231
Nov 03 15:46:51.524201 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:51.524177    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"]
Nov 03 15:46:52.544015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:52.543032    4955 generic.go:356] "Generic (PLEG): container finished" podID="19731d76-6fcb-4e10-af2a-39e3c0082b33" containerID="a30403f28c0ac63bfbb4bd2f33319829d0c13c1b11ba996181fa4509e159ef7a" exitCode=0
Nov 03 15:46:52.544015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:52.543170    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f" event={"ID":"19731d76-6fcb-4e10-af2a-39e3c0082b33","Type":"ContainerDied","Data":"a30403f28c0ac63bfbb4bd2f33319829d0c13c1b11ba996181fa4509e159ef7a"}
Nov 03 15:46:52.544015 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:52.543222    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f" event={"ID":"19731d76-6fcb-4e10-af2a-39e3c0082b33","Type":"ContainerStarted","Data":"0301d75759be5e328b844c1bd8aa8365b50b43782c0ea15dab29f26247526231"}
Nov 03 15:46:53.696759 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.696729    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:53.739521 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.739485    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca\") pod \"19731d76-6fcb-4e10-af2a-39e3c0082b33\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") "
Nov 03 15:46:53.739695 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.739548    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn\" (UniqueName: \"kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn\") pod \"19731d76-6fcb-4e10-af2a-39e3c0082b33\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") "
Nov 03 15:46:53.739695 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.739595    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca\") pod \"19731d76-6fcb-4e10-af2a-39e3c0082b33\" (UID: \"19731d76-6fcb-4e10-af2a-39e3c0082b33\") "
Nov 03 15:46:53.739920 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.739897    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca" pod "19731d76-6fcb-4e10-af2a-39e3c0082b33" (UID: "19731d76-6fcb-4e10-af2a-39e3c0082b33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:46:53.740053 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.740039    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-openshift-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:53.741850 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.741827    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca" pod "19731d76-6fcb-4e10-af2a-39e3c0082b33" (UID: "19731d76-6fcb-4e10-af2a-39e3c0082b33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:53.741928 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.741896    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn" pod "19731d76-6fcb-4e10-af2a-39e3c0082b33" (UID: "19731d76-6fcb-4e10-af2a-39e3c0082b33"). InnerVolumeSpecName "kube-api-access-l4dxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:53.870040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.866362    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dxn\" (UniqueName: \"kubernetes.io/projected/19731d76-6fcb-4e10-af2a-39e3c0082b33-kube-api-access-l4dxn\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:53.870040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.866401    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19731d76-6fcb-4e10-af2a-39e3c0082b33-maas-api-xks-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:53.870040 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:53.868161    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:46:54.555501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:54.555473    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"
Nov 03 15:46:54.555689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:54.555511    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f" event={"ID":"19731d76-6fcb-4e10-af2a-39e3c0082b33","Type":"ContainerDied","Data":"0301d75759be5e328b844c1bd8aa8365b50b43782c0ea15dab29f26247526231"}
Nov 03 15:46:54.555689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:54.555559    4955 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0301d75759be5e328b844c1bd8aa8365b50b43782c0ea15dab29f26247526231"
Nov 03 15:46:57.938255 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:57.938203    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:46:58.280525 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.280448    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main" containerID="cri-o://2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715" gracePeriod=45
Nov 03 15:46:58.551376 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.551355    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:46:58.595124 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.595092    4955 generic.go:356] "Generic (PLEG): container finished" podID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerID="2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715" exitCode=0
Nov 03 15:46:58.595314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.595146    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"
Nov 03 15:46:58.595314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.595174    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" event={"ID":"7a26c442-d934-4dd7-a620-e547742d2cb1","Type":"ContainerDied","Data":"2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715"}
Nov 03 15:46:58.595314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.595218    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g" event={"ID":"7a26c442-d934-4dd7-a620-e547742d2cb1","Type":"ContainerDied","Data":"f9b651706e9270588b2baf29665328808fbaab2197c74d2492d6033b6d170baf"}
Nov 03 15:46:58.595314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.595238    4955 scope.go:122] "RemoveContainer" containerID="2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715"
Nov 03 15:46:58.603983 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.603952    4955 scope.go:122] "RemoveContainer" containerID="2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715"
Nov 03 15:46:58.604328 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:46:58.604306    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715\": container with ID starting with 2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715 not found: ID does not exist" containerID="2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715"
Nov 03 15:46:58.604389 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.604339    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715"} err="failed to get container status \"2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715\": rpc error: code = NotFound desc = could not find container \"2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715\": container with ID starting with 2c69a02c73e0cb4168d37e3a024ea730b556ae42552f4ed90c09078ee0bfd715 not found: ID does not exist"
Nov 03 15:46:58.629916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.629889    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home\") pod \"7a26c442-d934-4dd7-a620-e547742d2cb1\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") "
Nov 03 15:46:58.630094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.629938    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm\") pod \"7a26c442-d934-4dd7-a620-e547742d2cb1\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") "
Nov 03 15:46:58.630094 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630000    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache\") pod \"7a26c442-d934-4dd7-a620-e547742d2cb1\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") "
Nov 03 15:46:58.630212 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630173    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6\" (UniqueName: \"kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6\") pod \"7a26c442-d934-4dd7-a620-e547742d2cb1\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") "
Nov 03 15:46:58.630285 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630215    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs\") pod \"7a26c442-d934-4dd7-a620-e547742d2cb1\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") "
Nov 03 15:46:58.630285 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630265    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir\") pod \"7a26c442-d934-4dd7-a620-e547742d2cb1\" (UID: \"7a26c442-d934-4dd7-a620-e547742d2cb1\") "
Nov 03 15:46:58.630387 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630171    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home" pod "7a26c442-d934-4dd7-a620-e547742d2cb1" (UID: "7a26c442-d934-4dd7-a620-e547742d2cb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:58.630387 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630329    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache" pod "7a26c442-d934-4dd7-a620-e547742d2cb1" (UID: "7a26c442-d934-4dd7-a620-e547742d2cb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:58.630787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630701    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:58.630787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630725    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:58.630787 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.630732    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir" pod "7a26c442-d934-4dd7-a620-e547742d2cb1" (UID: "7a26c442-d934-4dd7-a620-e547742d2cb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:58.632623 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.632601    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs" pod "7a26c442-d934-4dd7-a620-e547742d2cb1" (UID: "7a26c442-d934-4dd7-a620-e547742d2cb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:46:58.632860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.632835    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm" pod "7a26c442-d934-4dd7-a620-e547742d2cb1" (UID: "7a26c442-d934-4dd7-a620-e547742d2cb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:46:58.632860 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.632838    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6" pod "7a26c442-d934-4dd7-a620-e547742d2cb1" (UID: "7a26c442-d934-4dd7-a620-e547742d2cb1"). InnerVolumeSpecName "kube-api-access-779v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:46:58.731755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.731719    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779v6\" (UniqueName: \"kubernetes.io/projected/7a26c442-d934-4dd7-a620-e547742d2cb1-kube-api-access-779v6\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:58.731755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.731750    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a26c442-d934-4dd7-a620-e547742d2cb1-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:58.731755 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.731760    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:58.732086 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:58.731768    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a26c442-d934-4dd7-a620-e547742d2cb1-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:46:59.052414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:59.052381    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"]
Nov 03 15:46:59.075505 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:59.075468    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e62ff3/distinct-2-w0-e62ff3-kserve-557bf46ff8-jmd4g"]
Nov 03 15:46:59.205907 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:46:59.205831    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" path="/var/lib/kubelet/pods/7a26c442-d934-4dd7-a620-e547742d2cb1/volumes"
Nov 03 15:47:02.286199 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:02.286162    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"]
Nov 03 15:47:02.286693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:02.286366    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" containerID="cri-o://b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d" gracePeriod=30
Nov 03 15:47:03.101802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.101763    4955 patch_prober.go:29] interesting pod/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:47:03.102002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.101814    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:47:03.299582 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.299521    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main" containerID="cri-o://a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f" gracePeriod=45
Nov 03 15:47:03.564209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.564174    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:47:03.625648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.625616    4955 generic.go:356] "Generic (PLEG): container finished" podID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerID="a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f" exitCode=0
Nov 03 15:47:03.625841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.625683    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"
Nov 03 15:47:03.625841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.625704    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" event={"ID":"a617d72f-fd5b-4830-a2d1-97d589493bb7","Type":"ContainerDied","Data":"a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f"}
Nov 03 15:47:03.625841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.625752    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql" event={"ID":"a617d72f-fd5b-4830-a2d1-97d589493bb7","Type":"ContainerDied","Data":"f8b557ebd7d342235692f72427e580ce30f1d26ad0d210a9aad8f40c3965b4e3"}
Nov 03 15:47:03.625841 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.625777    4955 scope.go:122] "RemoveContainer" containerID="a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f"
Nov 03 15:47:03.634691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.634668    4955 scope.go:122] "RemoveContainer" containerID="a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f"
Nov 03 15:47:03.634955 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:03.634937    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f\": container with ID starting with a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f not found: ID does not exist" containerID="a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f"
Nov 03 15:47:03.635048 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.634994    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f"} err="failed to get container status \"a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f\": rpc error: code = NotFound desc = could not find container \"a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f\": container with ID starting with a691a608a22a07b96388edea9429e8bd11e0184424d0540948179cafbe7dd05f not found: ID does not exist"
Nov 03 15:47:03.694002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.693890    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir\") pod \"a617d72f-fd5b-4830-a2d1-97d589493bb7\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") "
Nov 03 15:47:03.694002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.693942    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb\" (UniqueName: \"kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb\") pod \"a617d72f-fd5b-4830-a2d1-97d589493bb7\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") "
Nov 03 15:47:03.694225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694067    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home\") pod \"a617d72f-fd5b-4830-a2d1-97d589493bb7\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") "
Nov 03 15:47:03.694225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694134    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs\") pod \"a617d72f-fd5b-4830-a2d1-97d589493bb7\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") "
Nov 03 15:47:03.694225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694195    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir" pod "a617d72f-fd5b-4830-a2d1-97d589493bb7" (UID: "a617d72f-fd5b-4830-a2d1-97d589493bb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:03.694225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694200    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache\") pod \"a617d72f-fd5b-4830-a2d1-97d589493bb7\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") "
Nov 03 15:47:03.694452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694282    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm\") pod \"a617d72f-fd5b-4830-a2d1-97d589493bb7\" (UID: \"a617d72f-fd5b-4830-a2d1-97d589493bb7\") "
Nov 03 15:47:03.694452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694378    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home" pod "a617d72f-fd5b-4830-a2d1-97d589493bb7" (UID: "a617d72f-fd5b-4830-a2d1-97d589493bb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:03.694757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694734    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:03.694757 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.694758    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:03.695262 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.695240    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache" pod "a617d72f-fd5b-4830-a2d1-97d589493bb7" (UID: "a617d72f-fd5b-4830-a2d1-97d589493bb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:03.697063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.697033    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb" pod "a617d72f-fd5b-4830-a2d1-97d589493bb7" (UID: "a617d72f-fd5b-4830-a2d1-97d589493bb7"). InnerVolumeSpecName "kube-api-access-bq6lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:03.697395 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.697360    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs" pod "a617d72f-fd5b-4830-a2d1-97d589493bb7" (UID: "a617d72f-fd5b-4830-a2d1-97d589493bb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:47:03.697567 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.697436    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm" pod "a617d72f-fd5b-4830-a2d1-97d589493bb7" (UID: "a617d72f-fd5b-4830-a2d1-97d589493bb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:03.796081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.796045    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a617d72f-fd5b-4830-a2d1-97d589493bb7-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:03.796081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.796080    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:03.796290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.796090    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a617d72f-fd5b-4830-a2d1-97d589493bb7-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:03.796290 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:03.796101    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6lb\" (UniqueName: \"kubernetes.io/projected/a617d72f-fd5b-4830-a2d1-97d589493bb7-kube-api-access-bq6lb\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:04.054871 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:04.054781    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"]
Nov 03 15:47:04.063945 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:04.063917    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bc1f07/premium-w4-bc1f07-kserve-57b4cb4fd9-w6rql"]
Nov 03 15:47:05.204661 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:05.204626    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" path="/var/lib/kubelet/pods/a617d72f-fd5b-4830-a2d1-97d589493bb7/volumes"
Nov 03 15:47:07.559320 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.559266    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:47:07.637499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637464    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637514    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637540    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637649    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637679 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637670    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637703    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637741    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637774    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.637833 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637799    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs\") pod \"49086954-ec28-4a57-8c3c-2c3fb185d67d\" (UID: \"49086954-ec28-4a57-8c3c-2c3fb185d67d\") "
Nov 03 15:47:07.638050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637836    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:07.638050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.637894    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:47:07.638050 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.638021    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:07.638202 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.638183    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:07.638381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.638201    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.638381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.638219    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49086954-ec28-4a57-8c3c-2c3fb185d67d-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.638381 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.638232    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.638583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.638427    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:07.639962 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.639943    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "kube-api-access-j78zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:07.640449 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.640428    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:47:07.640689 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.640669    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:07.640760 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.640674    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy" pod "49086954-ec28-4a57-8c3c-2c3fb185d67d" (UID: "49086954-ec28-4a57-8c3c-2c3fb185d67d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:07.650054 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.650026    4955 generic.go:356] "Generic (PLEG): container finished" podID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerID="b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d" exitCode=0
Nov 03 15:47:07.650160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.650065    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" event={"ID":"49086954-ec28-4a57-8c3c-2c3fb185d67d","Type":"ContainerDied","Data":"b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d"}
Nov 03 15:47:07.650160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.650081    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"
Nov 03 15:47:07.650160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.650113    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws" event={"ID":"49086954-ec28-4a57-8c3c-2c3fb185d67d","Type":"ContainerDied","Data":"b9595708facb33d9631b788ea8fb17016630cad72cfd72a323d9a520063fc828"}
Nov 03 15:47:07.650160 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.650135    4955 scope.go:122] "RemoveContainer" containerID="b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d"
Nov 03 15:47:07.668142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.668121    4955 scope.go:122] "RemoveContainer" containerID="b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d"
Nov 03 15:47:07.668524 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:07.668482    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d\": container with ID starting with b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d not found: ID does not exist" containerID="b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d"
Nov 03 15:47:07.668593 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.668536    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d"} err="failed to get container status \"b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d\": rpc error: code = NotFound desc = could not find container \"b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d\": container with ID starting with b4e7194931975cd5aee49fd43b3477cc0e07f7fe99285311e22c3134e27d5a5d not found: ID does not exist"
Nov 03 15:47:07.739688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.739647    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j78zf\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-kube-api-access-j78zf\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.739688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.739682    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.739688 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.739692    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.739916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.739701    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49086954-ec28-4a57-8c3c-2c3fb185d67d-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.739916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.739710    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.739916 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.739719    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49086954-ec28-4a57-8c3c-2c3fb185d67d-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:07.848034 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.847908    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"]
Nov 03 15:47:07.852049 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:07.852017    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9855ff-openshift-default-67c459f57c-jj7ws"]
Nov 03 15:47:09.205168 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:09.205138    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" path="/var/lib/kubelet/pods/49086954-ec28-4a57-8c3c-2c3fb185d67d/volumes"
Nov 03 15:47:10.594319 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.594280    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"]
Nov 03 15:47:10.594747 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.594502    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" containerID="cri-o://0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d" gracePeriod=30
Nov 03 15:47:10.615105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.615075    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"]
Nov 03 15:47:10.615308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.615285    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" containerID="cri-o://02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962" gracePeriod=30
Nov 03 15:47:10.639981 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.639934    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"]
Nov 03 15:47:10.640157 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.640137    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" containerID="cri-o://5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca" gracePeriod=30
Nov 03 15:47:10.644805 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.644775    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"]
Nov 03 15:47:10.645006 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:10.644962    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" containerID="cri-o://914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863" gracePeriod=30
Nov 03 15:47:11.215409 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:11.215376    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:47:11.215600 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:11.215429    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:47:13.521800 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:13.521767    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:47:13.522220 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:13.521834    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:47:14.246979 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:14.246928    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:47:14.247171 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:14.247014    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:47:14.262234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:14.262199    4955 patch_prober.go:29] interesting pod/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:47:14.262383 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:14.262266    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:47:15.941684 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:15.941367    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:47:15.953906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:15.953877    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:47:16.006268 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.006231    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:47:16.014155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.014131    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:47:16.038566 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038533    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.038753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038588    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.038753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038623    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.038753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038660    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.038753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038696    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.038753 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038723    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038765    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038792    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038827    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038866    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038895    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038918    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038927    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.038963    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:47:16.039071 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039054    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039091    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039125    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039165    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039195    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039231    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039249    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039272    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039334    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039372    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039411    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039442    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039471    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039510    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039540    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039590    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039601    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039621    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039653    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.039698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039693    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039746    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039781    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo\") pod \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\" (UID: \"3b5534d9-f720-4262-a098-c5c8d9b3ff9e\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039809    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket\") pod \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\" (UID: \"295300b9-1fc6-40af-a64e-fbf48c8c3bc6\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039844    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data\") pod \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\" (UID: \"06a2dec3-269a-4bd3-831c-b1be9dfa8359\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039872    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039900    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.039942    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert\") pod \"d463324a-0f30-4de2-b492-77adaf702b09\" (UID: \"d463324a-0f30-4de2-b492-77adaf702b09\") "
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040054    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040522    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040544    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040561    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040575    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040588    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.040737 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040605    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.041499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040892    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.040938    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:47:16.041499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041131    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041499 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041234    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041490    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041582    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:47:16.041706 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041589    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041746    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041753    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041829    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041836    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.041869 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.041839    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.042370    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.042549    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.042906    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.042962    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.043083    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.043133    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.043405    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.043524 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.043474    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.044407 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.044353    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.044559 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.044527    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "kube-api-access-q5swp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.044648 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.044628    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "kube-api-access-rgw7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.045209 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.045171    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token" pod "295300b9-1fc6-40af-a64e-fbf48c8c3bc6" (UID: "295300b9-1fc6-40af-a64e-fbf48c8c3bc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.045448 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.045408    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:47:16.045550 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.045506    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.045816 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.045800    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "kube-api-access-4lqqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.046097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.046076    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo" pod "3b5534d9-f720-4262-a098-c5c8d9b3ff9e" (UID: "3b5534d9-f720-4262-a098-c5c8d9b3ff9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:47:16.046494 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.046477    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy" pod "06a2dec3-269a-4bd3-831c-b1be9dfa8359" (UID: "06a2dec3-269a-4bd3-831c-b1be9dfa8359"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:47:16.046616 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.046602    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c" pod "d463324a-0f30-4de2-b492-77adaf702b09" (UID: "d463324a-0f30-4de2-b492-77adaf702b09"). InnerVolumeSpecName "kube-api-access-rv26c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:16.142234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142201    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142234    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142257    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142234 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142272    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142285    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d463324a-0f30-4de2-b492-77adaf702b09-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142299    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142312    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142324    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142337    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142350    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142361    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142375    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d463324a-0f30-4de2-b492-77adaf702b09-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142388    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5swp\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-kube-api-access-q5swp\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142404    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142415    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142423    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142435    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142443    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142451    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lqqb\" (UniqueName: \"kubernetes.io/projected/06a2dec3-269a-4bd3-831c-b1be9dfa8359-kube-api-access-4lqqb\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142460    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142468    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142478    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv26c\" (UniqueName: \"kubernetes.io/projected/d463324a-0f30-4de2-b492-77adaf702b09-kube-api-access-rv26c\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142492    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142504    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgw7c\" (UniqueName: \"kubernetes.io/projected/3b5534d9-f720-4262-a098-c5c8d9b3ff9e-kube-api-access-rgw7c\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142516    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d463324a-0f30-4de2-b492-77adaf702b09-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142524    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142532    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a2dec3-269a-4bd3-831c-b1be9dfa8359-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142542    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a2dec3-269a-4bd3-831c-b1be9dfa8359-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142550    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.142845 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.142560    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295300b9-1fc6-40af-a64e-fbf48c8c3bc6-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:16.699997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.699936    4955 generic.go:356] "Generic (PLEG): container finished" podID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerID="5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca" exitCode=0
Nov 03 15:47:16.700173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.700016    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"
Nov 03 15:47:16.700173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.700029    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" event={"ID":"295300b9-1fc6-40af-a64e-fbf48c8c3bc6","Type":"ContainerDied","Data":"5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca"}
Nov 03 15:47:16.700173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.700085    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj" event={"ID":"295300b9-1fc6-40af-a64e-fbf48c8c3bc6","Type":"ContainerDied","Data":"b75e86d4e8517fbc9ff7df0fe1d70af252ffa3c96cd0defd7702796105034cfc"}
Nov 03 15:47:16.700173 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.700110    4955 scope.go:122] "RemoveContainer" containerID="5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca"
Nov 03 15:47:16.701370 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.701350    4955 generic.go:356] "Generic (PLEG): container finished" podID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerID="0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d" exitCode=0
Nov 03 15:47:16.701450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.701401    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"
Nov 03 15:47:16.701450 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.701408    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" event={"ID":"3b5534d9-f720-4262-a098-c5c8d9b3ff9e","Type":"ContainerDied","Data":"0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d"}
Nov 03 15:47:16.701562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.701453    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv" event={"ID":"3b5534d9-f720-4262-a098-c5c8d9b3ff9e","Type":"ContainerDied","Data":"9a536f0376e1810d0a3bcc39836151d2bd8fda9e2927ae6ccebc94a8d512bbc3"}
Nov 03 15:47:16.703243 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.703207    4955 generic.go:356] "Generic (PLEG): container finished" podID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerID="914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863" exitCode=0
Nov 03 15:47:16.703360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.703268    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"
Nov 03 15:47:16.703360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.703287    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" event={"ID":"06a2dec3-269a-4bd3-831c-b1be9dfa8359","Type":"ContainerDied","Data":"914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863"}
Nov 03 15:47:16.703360 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.703334    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9" event={"ID":"06a2dec3-269a-4bd3-831c-b1be9dfa8359","Type":"ContainerDied","Data":"e44f905bc049eecd09e06f43c16cc3d84e413acbaa38c7691253b8cc1f5847e2"}
Nov 03 15:47:16.704961 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.704934    4955 generic.go:356] "Generic (PLEG): container finished" podID="d463324a-0f30-4de2-b492-77adaf702b09" containerID="02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962" exitCode=0
Nov 03 15:47:16.705108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.705004    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" event={"ID":"d463324a-0f30-4de2-b492-77adaf702b09","Type":"ContainerDied","Data":"02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962"}
Nov 03 15:47:16.705108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.705070    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7" event={"ID":"d463324a-0f30-4de2-b492-77adaf702b09","Type":"ContainerDied","Data":"2b790eb96b3afe3b155f3adef5b5a3fdc69ddd166a47c8eea102c30367ee640f"}
Nov 03 15:47:16.705108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.705017    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"
Nov 03 15:47:16.711580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.711547    4955 scope.go:122] "RemoveContainer" containerID="5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca"
Nov 03 15:47:16.711882 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:16.711854    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca\": container with ID starting with 5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca not found: ID does not exist" containerID="5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca"
Nov 03 15:47:16.712129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.711893    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca"} err="failed to get container status \"5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca\": rpc error: code = NotFound desc = could not find container \"5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca\": container with ID starting with 5b69d5143c126a889fd271b2bfdd73d382a7da862a7042d8f3b801380be927ca not found: ID does not exist"
Nov 03 15:47:16.712129 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.711915    4955 scope.go:122] "RemoveContainer" containerID="0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d"
Nov 03 15:47:16.721128 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.721110    4955 scope.go:122] "RemoveContainer" containerID="0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d"
Nov 03 15:47:16.721402 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:16.721383    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d\": container with ID starting with 0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d not found: ID does not exist" containerID="0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d"
Nov 03 15:47:16.721489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.721411    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d"} err="failed to get container status \"0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d\": rpc error: code = NotFound desc = could not find container \"0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d\": container with ID starting with 0afe162eb01f08db32697ae6437bceb98a20b3835e88ddde879f7668d214301d not found: ID does not exist"
Nov 03 15:47:16.721489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.721430    4955 scope.go:122] "RemoveContainer" containerID="914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863"
Nov 03 15:47:16.729808 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.729791    4955 scope.go:122] "RemoveContainer" containerID="914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863"
Nov 03 15:47:16.730097 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:16.730075    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863\": container with ID starting with 914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863 not found: ID does not exist" containerID="914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863"
Nov 03 15:47:16.730169 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.730110    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863"} err="failed to get container status \"914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863\": rpc error: code = NotFound desc = could not find container \"914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863\": container with ID starting with 914410d95aab8b4954ccdc0e4ffc29a274083a03aacb992a34eeb35595279863 not found: ID does not exist"
Nov 03 15:47:16.730169 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.730134    4955 scope.go:122] "RemoveContainer" containerID="02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962"
Nov 03 15:47:16.738144 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.738120    4955 scope.go:122] "RemoveContainer" containerID="02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962"
Nov 03 15:47:16.738425 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:16.738405    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962\": container with ID starting with 02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962 not found: ID does not exist" containerID="02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962"
Nov 03 15:47:16.738472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.738430    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962"} err="failed to get container status \"02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962\": rpc error: code = NotFound desc = could not find container \"02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962\": container with ID starting with 02fd74461283d68126166bd719a962380096da1eddcd2cece4b29d13b1d07962 not found: ID does not exist"
Nov 03 15:47:16.809038 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.809000    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"]
Nov 03 15:47:16.809270 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.809246    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" podUID="48814361-2d4f-4e23-b7f0-065c7cae298a" containerName="maas-api" containerID="cri-o://aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034" gracePeriod=30
Nov 03 15:47:16.888487 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.888399    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"]
Nov 03 15:47:16.897894 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:16.897859    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-ltnb7"]
Nov 03 15:47:17.010905 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.010854    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"]
Nov 03 15:47:17.043992 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.038082    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-fzwh9"]
Nov 03 15:47:17.076767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.076743    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:47:17.152659 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.152631    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm\" (UniqueName: \"kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm\") pod \"48814361-2d4f-4e23-b7f0-065c7cae298a\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") "
Nov 03 15:47:17.152826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.152674    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca\") pod \"48814361-2d4f-4e23-b7f0-065c7cae298a\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") "
Nov 03 15:47:17.152826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.152722    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\") pod \"48814361-2d4f-4e23-b7f0-065c7cae298a\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") "
Nov 03 15:47:17.152826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.152787    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs\") pod \"48814361-2d4f-4e23-b7f0-065c7cae298a\" (UID: \"48814361-2d4f-4e23-b7f0-065c7cae298a\") "
Nov 03 15:47:17.153145 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.153112    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca" pod "48814361-2d4f-4e23-b7f0-065c7cae298a" (UID: "48814361-2d4f-4e23-b7f0-065c7cae298a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:47:17.155128 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.155099    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs" pod "48814361-2d4f-4e23-b7f0-065c7cae298a" (UID: "48814361-2d4f-4e23-b7f0-065c7cae298a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:47:17.155249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.155156    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls" pod "48814361-2d4f-4e23-b7f0-065c7cae298a" (UID: "48814361-2d4f-4e23-b7f0-065c7cae298a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:47:17.155249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.155187    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm" pod "48814361-2d4f-4e23-b7f0-065c7cae298a" (UID: "48814361-2d4f-4e23-b7f0-065c7cae298a"). InnerVolumeSpecName "kube-api-access-mvtfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:47:17.159853 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.159830    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"]
Nov 03 15:47:17.169227 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.169205    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-4ndfv"]
Nov 03 15:47:17.209061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.208998    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" path="/var/lib/kubelet/pods/06a2dec3-269a-4bd3-831c-b1be9dfa8359/volumes"
Nov 03 15:47:17.209541 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.209525    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" path="/var/lib/kubelet/pods/3b5534d9-f720-4262-a098-c5c8d9b3ff9e/volumes"
Nov 03 15:47:17.210003 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.209980    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d463324a-0f30-4de2-b492-77adaf702b09" path="/var/lib/kubelet/pods/d463324a-0f30-4de2-b492-77adaf702b09/volumes"
Nov 03 15:47:17.254491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.254459    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:17.254491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.254486    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtfm\" (UniqueName: \"kubernetes.io/projected/48814361-2d4f-4e23-b7f0-065c7cae298a-kube-api-access-mvtfm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:17.254491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.254498    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48814361-2d4f-4e23-b7f0-065c7cae298a-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:17.254720 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.254506    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48814361-2d4f-4e23-b7f0-065c7cae298a-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:47:17.289178 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.289147    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"]
Nov 03 15:47:17.295651 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.295622    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cad072f-openshift-default-64db96fb4f-jj4tj"]
Nov 03 15:47:17.712807 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.712776    4955 generic.go:356] "Generic (PLEG): container finished" podID="48814361-2d4f-4e23-b7f0-065c7cae298a" containerID="aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034" exitCode=0
Nov 03 15:47:17.713024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.712826    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"
Nov 03 15:47:17.713024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.712859    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" event={"ID":"48814361-2d4f-4e23-b7f0-065c7cae298a","Type":"ContainerDied","Data":"aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034"}
Nov 03 15:47:17.713024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.712911    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d" event={"ID":"48814361-2d4f-4e23-b7f0-065c7cae298a","Type":"ContainerDied","Data":"b9b3a9f23a5ef0446683b1904b0846b3b111758fcbf38c154db53b2675df0559"}
Nov 03 15:47:17.713024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.712939    4955 scope.go:122] "RemoveContainer" containerID="aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034"
Nov 03 15:47:17.721950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.721932    4955 scope.go:122] "RemoveContainer" containerID="aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034"
Nov 03 15:47:17.722252 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:17.722223    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034\": container with ID starting with aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034 not found: ID does not exist" containerID="aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034"
Nov 03 15:47:17.722308 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.722261    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034"} err="failed to get container status \"aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034\": rpc error: code = NotFound desc = could not find container \"aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034\": container with ID starting with aeea4d0bc4a5c31b58eed1aef612723a4642ed505cbfd0d20a70e4527af3a034 not found: ID does not exist"
Nov 03 15:47:17.885106 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.885071    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"]
Nov 03 15:47:17.890249 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:17.890213    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bc1f07-547f4b4fb8-cgx9d"]
Nov 03 15:47:19.204374 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:19.204343    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" path="/var/lib/kubelet/pods/295300b9-1fc6-40af-a64e-fbf48c8c3bc6/volumes"
Nov 03 15:47:19.204941 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:19.204919    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48814361-2d4f-4e23-b7f0-065c7cae298a" path="/var/lib/kubelet/pods/48814361-2d4f-4e23-b7f0-065c7cae298a/volumes"
Nov 03 15:47:27.938951 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:27.938907    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:47:32.460592 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:47:32.460555    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:47:57.939076 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:47:57.939038    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:48:05.808954 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:05.808926    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:48:05.817599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:05.817572    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:48:05.820215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:05.820197    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:48:05.828215 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:05.828197    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:48:25.426995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.426880    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"]
Nov 03 15:48:25.450995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.427619    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"]
Nov 03 15:48:25.650142 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.650100    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:48:25.653485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.653447    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:48:25.740280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.740066    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:48:25.740280 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.740136    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:48:25.837776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.832119    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:48:25.837776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:25.832206    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:48:27.938858 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:48:27.938819    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:48:40.659816 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:40.659776    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main" containerID="cri-o://34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979" gracePeriod=45
Nov 03 15:48:40.661799 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:40.661728    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main" containerID="cri-o://1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09" gracePeriod=45
Nov 03 15:48:40.942839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:40.942814    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:48:40.945738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:40.945716    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:48:41.022033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.021992    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm\") pod \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") "
Nov 03 15:48:41.022033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022036    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir\") pod \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") "
Nov 03 15:48:41.022326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022078    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh\" (UniqueName: \"kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh\") pod \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") "
Nov 03 15:48:41.022326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022145    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs\") pod \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") "
Nov 03 15:48:41.022326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022174    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs\") pod \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") "
Nov 03 15:48:41.022326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022195    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home\") pod \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") "
Nov 03 15:48:41.022326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022233    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache\") pod \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") "
Nov 03 15:48:41.022608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022372    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir" pod "f1c1e120-a539-403d-a6fd-851d2ffa4abc" (UID: "f1c1e120-a539-403d-a6fd-851d2ffa4abc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.022608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022434    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir\") pod \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\" (UID: \"e33ccc80-ef0b-413c-bef5-8b605a025ecc\") "
Nov 03 15:48:41.022608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022504    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home" pod "e33ccc80-ef0b-413c-bef5-8b605a025ecc" (UID: "e33ccc80-ef0b-413c-bef5-8b605a025ecc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.022608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022521    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs\" (UniqueName: \"kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs\") pod \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") "
Nov 03 15:48:41.022608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022529    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache" pod "e33ccc80-ef0b-413c-bef5-8b605a025ecc" (UID: "e33ccc80-ef0b-413c-bef5-8b605a025ecc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.022608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022585    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm\") pod \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") "
Nov 03 15:48:41.022903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022632    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache\") pod \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") "
Nov 03 15:48:41.022903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022671    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir" pod "e33ccc80-ef0b-413c-bef5-8b605a025ecc" (UID: "e33ccc80-ef0b-413c-bef5-8b605a025ecc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.022903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.022685    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home\") pod \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\" (UID: \"f1c1e120-a539-403d-a6fd-851d2ffa4abc\") "
Nov 03 15:48:41.023134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.023082    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.023134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.023105    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.023134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.023119    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.023134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.023132    4955 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tmp-dir\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.023393 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.023368    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home" pod "f1c1e120-a539-403d-a6fd-851d2ffa4abc" (UID: "f1c1e120-a539-403d-a6fd-851d2ffa4abc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.023547 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.023523    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache" pod "f1c1e120-a539-403d-a6fd-851d2ffa4abc" (UID: "f1c1e120-a539-403d-a6fd-851d2ffa4abc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.025424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.025272    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs" pod "f1c1e120-a539-403d-a6fd-851d2ffa4abc" (UID: "f1c1e120-a539-403d-a6fd-851d2ffa4abc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:48:41.025424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.025299    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm" pod "e33ccc80-ef0b-413c-bef5-8b605a025ecc" (UID: "e33ccc80-ef0b-413c-bef5-8b605a025ecc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.025424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.025312    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh" pod "e33ccc80-ef0b-413c-bef5-8b605a025ecc" (UID: "e33ccc80-ef0b-413c-bef5-8b605a025ecc"). InnerVolumeSpecName "kube-api-access-nn9rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:48:41.025424 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.025382    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs" pod "f1c1e120-a539-403d-a6fd-851d2ffa4abc" (UID: "f1c1e120-a539-403d-a6fd-851d2ffa4abc"). InnerVolumeSpecName "kube-api-access-g6rbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:48:41.025622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.025444    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs" pod "e33ccc80-ef0b-413c-bef5-8b605a025ecc" (UID: "e33ccc80-ef0b-413c-bef5-8b605a025ecc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:48:41.025622 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.025512    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm" pod "f1c1e120-a539-403d-a6fd-851d2ffa4abc" (UID: "f1c1e120-a539-403d-a6fd-851d2ffa4abc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:48:41.123605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123563    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9rh\" (UniqueName: \"kubernetes.io/projected/e33ccc80-ef0b-413c-bef5-8b605a025ecc-kube-api-access-nn9rh\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123597    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33ccc80-ef0b-413c-bef5-8b605a025ecc-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123611    4955 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c1e120-a539-403d-a6fd-851d2ffa4abc-tls-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123623    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rbs\" (UniqueName: \"kubernetes.io/projected/f1c1e120-a539-403d-a6fd-851d2ffa4abc-kube-api-access-g6rbs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123636    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123647    4955 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-model-cache\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123658    4955 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c1e120-a539-403d-a6fd-851d2ffa4abc-home\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.123851 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.123669    4955 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33ccc80-ef0b-413c-bef5-8b605a025ecc-dshm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:41.355495 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.355462    4955 generic.go:356] "Generic (PLEG): container finished" podID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerID="34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979" exitCode=0
Nov 03 15:48:41.355682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.355512    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"
Nov 03 15:48:41.355682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.355537    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" event={"ID":"e33ccc80-ef0b-413c-bef5-8b605a025ecc","Type":"ContainerDied","Data":"34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979"}
Nov 03 15:48:41.355682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.355572    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr" event={"ID":"e33ccc80-ef0b-413c-bef5-8b605a025ecc","Type":"ContainerDied","Data":"ba8dd715d46d1f79fe0036ee3d5099b7bf004117cf535003a119d1bb6479298c"}
Nov 03 15:48:41.355682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.355589    4955 scope.go:122] "RemoveContainer" containerID="34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979"
Nov 03 15:48:41.357155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.357129    4955 generic.go:356] "Generic (PLEG): container finished" podID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerID="1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09" exitCode=0
Nov 03 15:48:41.357259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.357169    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"
Nov 03 15:48:41.357259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.357216    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" event={"ID":"f1c1e120-a539-403d-a6fd-851d2ffa4abc","Type":"ContainerDied","Data":"1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09"}
Nov 03 15:48:41.357259 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.357251    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr" event={"ID":"f1c1e120-a539-403d-a6fd-851d2ffa4abc","Type":"ContainerDied","Data":"a026d3650c6b325fff33a7329689d6ca0f7986987236100ac15e64dda6cd3bb2"}
Nov 03 15:48:41.366982 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.366946    4955 scope.go:122] "RemoveContainer" containerID="34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979"
Nov 03 15:48:41.367318 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:48:41.367299    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979\": container with ID starting with 34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979 not found: ID does not exist" containerID="34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979"
Nov 03 15:48:41.367430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.367326    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979"} err="failed to get container status \"34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979\": rpc error: code = NotFound desc = could not find container \"34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979\": container with ID starting with 34bbf35c7820b77f2474a440312c1d9d0479b366a2cd6b7603e1ea926aaed979 not found: ID does not exist"
Nov 03 15:48:41.367430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.367346    4955 scope.go:122] "RemoveContainer" containerID="1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09"
Nov 03 15:48:41.375271 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.375253    4955 scope.go:122] "RemoveContainer" containerID="1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09"
Nov 03 15:48:41.375542 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:48:41.375523    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09\": container with ID starting with 1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09 not found: ID does not exist" containerID="1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09"
Nov 03 15:48:41.375607 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.375548    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09"} err="failed to get container status \"1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09\": rpc error: code = NotFound desc = could not find container \"1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09\": container with ID starting with 1c163cefdbd22e507765254a648b445f4bf3a13d9653290cce094fbda8473c09 not found: ID does not exist"
Nov 03 15:48:41.522715 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.522632    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"]
Nov 03 15:48:41.529421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.529393    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/simulator-w3-8b841a-kserve-d9db7d79d-sv9qr"]
Nov 03 15:48:41.644273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.644238    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"]
Nov 03 15:48:41.651932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:41.651901    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b841a/premium-w3-8b841a-kserve-645fdc47cf-cw8nr"]
Nov 03 15:48:43.205295 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:43.205260    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" path="/var/lib/kubelet/pods/e33ccc80-ef0b-413c-bef5-8b605a025ecc/volumes"
Nov 03 15:48:43.205691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:43.205654    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" path="/var/lib/kubelet/pods/f1c1e120-a539-403d-a6fd-851d2ffa4abc/volumes"
Nov 03 15:48:53.968273 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:53.968235    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:48:54.241228 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.241133    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"]
Nov 03 15:48:54.241414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.241351    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx" podUID="71e6c823-1909-45a1-9fba-ce066d8ef5bb" containerName="maas-api" containerID="cri-o://0a218f1e7b12fbc6379068b5bdf5550e5b101f0ea12be02e4fa27c59e59964d5" gracePeriod=30
Nov 03 15:48:54.433485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.432591    4955 generic.go:356] "Generic (PLEG): container finished" podID="71e6c823-1909-45a1-9fba-ce066d8ef5bb" containerID="0a218f1e7b12fbc6379068b5bdf5550e5b101f0ea12be02e4fa27c59e59964d5" exitCode=0
Nov 03 15:48:54.433485 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.432646    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx" event={"ID":"71e6c823-1909-45a1-9fba-ce066d8ef5bb","Type":"ContainerDied","Data":"0a218f1e7b12fbc6379068b5bdf5550e5b101f0ea12be02e4fa27c59e59964d5"}
Nov 03 15:48:54.548795 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.548767    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:48:54.664401 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.664356    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca\") pod \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") "
Nov 03 15:48:54.664557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.664416    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs\") pod \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") "
Nov 03 15:48:54.664557 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.664480    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls\") pod \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") "
Nov 03 15:48:54.664691 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.664666    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg\" (UniqueName: \"kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg\") pod \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\" (UID: \"71e6c823-1909-45a1-9fba-ce066d8ef5bb\") "
Nov 03 15:48:54.664814 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.664793    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca" pod "71e6c823-1909-45a1-9fba-ce066d8ef5bb" (UID: "71e6c823-1909-45a1-9fba-ce066d8ef5bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:48:54.665194 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.665173    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e6c823-1909-45a1-9fba-ce066d8ef5bb-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:54.666708 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.666687    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs" pod "71e6c823-1909-45a1-9fba-ce066d8ef5bb" (UID: "71e6c823-1909-45a1-9fba-ce066d8ef5bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:48:54.666791 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.666761    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls" pod "71e6c823-1909-45a1-9fba-ce066d8ef5bb" (UID: "71e6c823-1909-45a1-9fba-ce066d8ef5bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:48:54.666867 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.666844    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg" pod "71e6c823-1909-45a1-9fba-ce066d8ef5bb" (UID: "71e6c823-1909-45a1-9fba-ce066d8ef5bb"). InnerVolumeSpecName "kube-api-access-xwcbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:48:54.765917 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.765873    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:54.765917 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.765911    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcbg\" (UniqueName: \"kubernetes.io/projected/71e6c823-1909-45a1-9fba-ce066d8ef5bb-kube-api-access-xwcbg\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:54.765917 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:54.765921    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e6c823-1909-45a1-9fba-ce066d8ef5bb-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:48:55.438500 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:55.438463    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx" event={"ID":"71e6c823-1909-45a1-9fba-ce066d8ef5bb","Type":"ContainerDied","Data":"5317e38cf9f8fd821ef8383e1edb14803728b6ff8c228474de902041fa47c0aa"}
Nov 03 15:48:55.438988 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:55.438517    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"
Nov 03 15:48:55.438988 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:55.438523    4955 scope.go:122] "RemoveContainer" containerID="0a218f1e7b12fbc6379068b5bdf5550e5b101f0ea12be02e4fa27c59e59964d5"
Nov 03 15:48:55.607834 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:55.607750    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"]
Nov 03 15:48:55.612489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:55.612460    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b841a-59b6d8584b-5nrtx"]
Nov 03 15:48:57.204080 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:48:57.204048    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e6c823-1909-45a1-9fba-ce066d8ef5bb" path="/var/lib/kubelet/pods/71e6c823-1909-45a1-9fba-ce066d8ef5bb/volumes"
Nov 03 15:48:57.938691 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:48:57.938646    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:49:09.902542 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:49:09.902495    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:49:27.938073 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:49:27.938035    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:49:57.938124 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:49:57.938088    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:50:27.938348 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:50:27.938309    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:50:38.248551 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:38.248510    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:50:40.619501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:40.619454    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:50:40.620239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:40.620201    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:50:40.730133 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:40.730090    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:50:40.730440 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:40.730409    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:50:40.830997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:40.830093    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:50:40.830997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:40.830180    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:50:57.937114 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:50:57.937075    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:50:59.993686 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:50:59.993650    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:51:10.914629 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:51:10.914597    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:51:27.938365 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:51:27.938324    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:51:35.177881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:51:35.177844    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:51:48.582606 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:51:48.582569    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:51:54.062150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:51:54.062117    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"]
Nov 03 15:51:54.112667 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:51:54.112632    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d29c20-46p7f"]
Nov 03 15:51:55.203932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:51:55.203893    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19731d76-6fcb-4e10-af2a-39e3c0082b33" path="/var/lib/kubelet/pods/19731d76-6fcb-4e10-af2a-39e3c0082b33/volumes"
Nov 03 15:51:57.937561 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:51:57.937519    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:52:19.299501 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:52:19.299463    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:52:27.938233 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:52:27.938193    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:52:30.055404 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:52:30.055360    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:52:42.847185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:52:42.847152    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:52:57.937582 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:52:57.937543    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:53:05.839074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:05.839044    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:53:05.847906 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:05.847877    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:53:05.850489 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:05.850465    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:53:05.858722 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:05.858701    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:53:06.429738 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:06.429708    4955 scope.go:122] "RemoveContainer" containerID="a30403f28c0ac63bfbb4bd2f33319829d0c13c1b11ba996181fa4509e159ef7a"
Nov 03 15:53:08.085490 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:08.085458    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:53:17.897529 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:17.897491    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:53:27.938505 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:53:27.938466    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:53:36.279336 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:36.279295    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:53:48.879196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:53:48.879119    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:53:57.938370 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:53:57.938333    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:54:08.879411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:54:08.879374    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:54:11.858002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:54:11.857115    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-5zkfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:54:11.858002 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:54:11.857222    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-5zkfh" podUID="932b46a7-66e5-4991-ae83-a6086edb8cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:54:27.937077 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:54:27.937034    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:54:30.897109 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:54:30.897072    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:54:43.973599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:54:43.973563    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:54:55.264172 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:54:55.264133    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:54:57.938528 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:54:57.938485    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:55:17.983427 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:55:17.983395    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:55:27.938279 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:55:27.938233    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:55:28.282042 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:55:28.281933    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:55:42.076107 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:55:42.076069    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:55:57.940255 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:55:57.940216    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:56:03.873619 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:03.873578    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:56:14.763155 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:14.763121    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:56:27.937624 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:56:27.937578    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:56:33.760921 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:33.760883    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:56:44.394480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:44.394398    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:56:51.598057 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598017    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"]
Nov 03 15:56:51.598624 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598607    4955 cpu_manager.go:414] "containerMap: removing container" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy"
Nov 03 15:56:51.598677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598627    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy"
Nov 03 15:56:51.598677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598642    4955 cpu_manager.go:414] "containerMap: removing container" podUID="48814361-2d4f-4e23-b7f0-065c7cae298a" containerName="maas-api"
Nov 03 15:56:51.598677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598650    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48814361-2d4f-4e23-b7f0-065c7cae298a" containerName="maas-api"
Nov 03 15:56:51.598677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598664    4955 cpu_manager.go:414] "containerMap: removing container" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main"
Nov 03 15:56:51.598677 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598673    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598682    4955 cpu_manager.go:414] "containerMap: removing container" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598690    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598707    4955 cpu_manager.go:414] "containerMap: removing container" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598714    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598727    4955 cpu_manager.go:414] "containerMap: removing container" podUID="19731d76-6fcb-4e10-af2a-39e3c0082b33" containerName="revoke-keys"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598734    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19731d76-6fcb-4e10-af2a-39e3c0082b33" containerName="revoke-keys"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598762    4955 cpu_manager.go:414] "containerMap: removing container" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598769    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598778    4955 cpu_manager.go:414] "containerMap: removing container" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598785    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598796    4955 cpu_manager.go:414] "containerMap: removing container" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598803    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598818    4955 cpu_manager.go:414] "containerMap: removing container" podUID="71e6c823-1909-45a1-9fba-ce066d8ef5bb" containerName="maas-api"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598824    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e6c823-1909-45a1-9fba-ce066d8ef5bb" containerName="maas-api"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598835    4955 cpu_manager.go:414] "containerMap: removing container" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598842    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598854    4955 cpu_manager.go:414] "containerMap: removing container" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy"
Nov 03 15:56:51.598865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.598861    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599123    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33ccc80-ef0b-413c-bef5-8b605a025ecc" containerName="main"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599142    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c1e120-a539-403d-a6fd-851d2ffa4abc" containerName="main"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599157    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="295300b9-1fc6-40af-a64e-fbf48c8c3bc6" containerName="istio-proxy"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599167    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="19731d76-6fcb-4e10-af2a-39e3c0082b33" containerName="revoke-keys"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599183    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="a617d72f-fd5b-4830-a2d1-97d589493bb7" containerName="main"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599193    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="d463324a-0f30-4de2-b492-77adaf702b09" containerName="istio-proxy"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599201    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="49086954-ec28-4a57-8c3c-2c3fb185d67d" containerName="istio-proxy"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599212    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e6c823-1909-45a1-9fba-ce066d8ef5bb" containerName="maas-api"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599235    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a26c442-d934-4dd7-a620-e547742d2cb1" containerName="main"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599243    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a2dec3-269a-4bd3-831c-b1be9dfa8359" containerName="istio-proxy"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599275    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5534d9-f720-4262-a098-c5c8d9b3ff9e" containerName="istio-proxy"
Nov 03 15:56:51.599599 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.599300    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="48814361-2d4f-4e23-b7f0-065c7cae298a" containerName="maas-api"
Nov 03 15:56:51.602861 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.602842    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.664670 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.664635    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.664865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.664685    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.664865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.664711    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.664865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.664807    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.664865 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.664857    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.665066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.664941    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.665066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.665000    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svjt\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.665066 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.665055    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.665169 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.665080    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.703170 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.703132    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"]
Nov 03 15:56:51.718035 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.716354    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d9d854-openshift-default-dockercfg-9skb7\""
Nov 03 15:56:51.766481 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766449    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766496    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766541    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766574    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766623    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766671    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766725    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766755    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svjt\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766785    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.766946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.766939    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.767196 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.767109    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.767252 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.767238    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.767580 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.767552    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.767715 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.767577    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.769143 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.769119    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.769311 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.769293    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.834044 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.834008    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.840644 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.840603    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svjt\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt\") pod \"e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") " pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:51.916214 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:51.916180    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:52.345480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:52.345451    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"]
Nov 03 15:56:52.347913 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:56:52.347887    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3804ce19_0c5d_4568_9a41_df8d8765b172.slice/crio-43e2588d1930ae680cefed4f34ae3a5a24065f91243d937409d4dfa798be3fa0 WatchSource:0}: Error finding container 43e2588d1930ae680cefed4f34ae3a5a24065f91243d937409d4dfa798be3fa0: Status 404 returned error can't find the container with id 43e2588d1930ae680cefed4f34ae3a5a24065f91243d937409d4dfa798be3fa0
Nov 03 15:56:52.350284 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:52.350249    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:56:52.350365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:52.350319    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:56:52.350365 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:52.350346    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:56:52.961188 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:52.961156    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" event={"ID":"3804ce19-0c5d-4568-9a41-df8d8765b172","Type":"ContainerStarted","Data":"b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"}
Nov 03 15:56:52.961608 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:52.961209    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" event={"ID":"3804ce19-0c5d-4568-9a41-df8d8765b172","Type":"ContainerStarted","Data":"43e2588d1930ae680cefed4f34ae3a5a24065f91243d937409d4dfa798be3fa0"}
Nov 03 15:56:53.086640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:53.086584    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podStartSLOduration=2.086569684 podStartE2EDuration="2.086569684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:56:53.084911685 +0000 UTC m=+2928.009729611" watchObservedRunningTime="2026-11-03 15:56:53.086569684 +0000 UTC m=+2928.011387597"
Nov 03 15:56:53.916935 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:53.916898    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:53.922063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:53.922031    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:53.967880 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:53.967844    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:53.969097 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:53.969074    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:56:57.937457 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:56:57.937422    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:56:59.720326 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.720283    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"]
Nov 03 15:56:59.725136 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.725113    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.743464 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.743436    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.743610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.743480    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsb7n\" (UniqueName: \"kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n\") pod \"payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.785922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.785891    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"]
Nov 03 15:56:59.833723 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.833693    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d9d854\""
Nov 03 15:56:59.851628 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.851588    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.851915 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.851884    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsb7n\" (UniqueName: \"kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n\") pod \"payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.868331 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.852584    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.887149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.879900    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"]
Nov 03 15:56:59.887149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.884644    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:56:59.914535 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.914505    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsb7n\" (UniqueName: \"kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n\") pod \"payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:56:59.928827 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.928795    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"]
Nov 03 15:56:59.939776 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.939749    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-056ea753-dockercfg-k7m6s\""
Nov 03 15:56:59.953342 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.953312    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz76v\" (UniqueName: \"kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v\") pod \"payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:56:59.953517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:56:59.953489    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:00.039649 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.039557    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:57:00.054902 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.054868    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:00.055185 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.055165    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz76v\" (UniqueName: \"kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v\") pod \"payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:00.056452 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.056430    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:00.120491 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.120426    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz76v\" (UniqueName: \"kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v\") pod \"payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:00.202884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.202839    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:00.434504 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.434472    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"]
Nov 03 15:57:00.437152 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:57:00.437128    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d82c8c4_2a5d_4c2b_9984_5dc07fff6302.slice/crio-c5c0c06b572b9d46c9bc255326ead32594d52c50ed6c3ffb59860c26e336a0d8 WatchSource:0}: Error finding container c5c0c06b572b9d46c9bc255326ead32594d52c50ed6c3ffb59860c26e336a0d8: Status 404 returned error can't find the container with id c5c0c06b572b9d46c9bc255326ead32594d52c50ed6c3ffb59860c26e336a0d8
Nov 03 15:57:00.575061 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:00.575037    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"]
Nov 03 15:57:00.577319 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:57:00.577290    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2f23bd_e165_4e32_bd81_2612317fa00a.slice/crio-91689a85eb1499c2dc818b010cf00384c66e6dcd73825bc771dfbb5f9a2c7d90 WatchSource:0}: Error finding container 91689a85eb1499c2dc818b010cf00384c66e6dcd73825bc771dfbb5f9a2c7d90: Status 404 returned error can't find the container with id 91689a85eb1499c2dc818b010cf00384c66e6dcd73825bc771dfbb5f9a2c7d90
Nov 03 15:57:01.001019 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.000957    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" event={"ID":"0e2f23bd-e165-4e32-bd81-2612317fa00a","Type":"ContainerStarted","Data":"fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6"}
Nov 03 15:57:01.001492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.001033    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" event={"ID":"0e2f23bd-e165-4e32-bd81-2612317fa00a","Type":"ContainerStarted","Data":"91689a85eb1499c2dc818b010cf00384c66e6dcd73825bc771dfbb5f9a2c7d90"}
Nov 03 15:57:01.001492 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.001164    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:01.002596 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.002573    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" event={"ID":"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302","Type":"ContainerStarted","Data":"f9f50e59785abd7cc203ea0dd656f3620ce46af9b57edd9962c35752c64822fc"}
Nov 03 15:57:01.002728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.002613    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:57:01.002728 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.002630    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" event={"ID":"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302","Type":"ContainerStarted","Data":"c5c0c06b572b9d46c9bc255326ead32594d52c50ed6c3ffb59860c26e336a0d8"}
Nov 03 15:57:01.119903 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.119844    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" podStartSLOduration=2.119829303 podStartE2EDuration="2.119829303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:57:01.118759344 +0000 UTC m=+2936.043577279" watchObservedRunningTime="2026-11-03 15:57:01.119829303 +0000 UTC m=+2936.044647216"
Nov 03 15:57:01.251932 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:01.251805    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" podStartSLOduration=2.251790271 podStartE2EDuration="2.251790271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:57:01.251296531 +0000 UTC m=+2936.176114445" watchObservedRunningTime="2026-11-03 15:57:01.251790271 +0000 UTC m=+2936.176608184"
Nov 03 15:57:02.025418 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.025384    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"]
Nov 03 15:57:02.028883 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.028865    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.075881 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.075837    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.076108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.075913    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.076108 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.075946    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khtpm\" (UniqueName: \"kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.076225 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.076136    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.087640 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.087608    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d9d854-serving-cert\""
Nov 03 15:57:02.139947 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.139907    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"]
Nov 03 15:57:02.177093 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.177057    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.177291 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.177114    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtpm\" (UniqueName: \"kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.177291 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.177230    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.177291 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:57:02.177235    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-serving-cert: secret "maas-api-e2e-worker-single-main-d9d854-serving-cert" not found
Nov 03 15:57:02.177467 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:57:02.177308    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls podName:65f48e48-a6b1-449e-b63b-44d9789304cf nodeName:}" failed. No retries permitted until 2026-11-03 15:57:02.677286392 +0000 UTC m=+2937.602104305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls") pod "maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" (UID: "65f48e48-a6b1-449e-b63b-44d9789304cf") : secret "maas-api-e2e-worker-single-main-d9d854-serving-cert" not found
Nov 03 15:57:02.177467 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.177364    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.178364 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.178326    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.179995 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.179954    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.242041 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.242011    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtpm\" (UniqueName: \"kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.682769 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.682732    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.685324 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.685297    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:02.942410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:02.942317    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:03.320296 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:03.320267    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"]
Nov 03 15:57:03.322739 ip-10-0-134-23 kubenswrapper[4955]: W1103 15:57:03.322702    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f48e48_a6b1_449e_b63b_44d9789304cf.slice/crio-15a815731387979106d134581bd4e8c69f7eb45e4a68e21d4754080fdf3723e1 WatchSource:0}: Error finding container 15a815731387979106d134581bd4e8c69f7eb45e4a68e21d4754080fdf3723e1: Status 404 returned error can't find the container with id 15a815731387979106d134581bd4e8c69f7eb45e4a68e21d4754080fdf3723e1
Nov 03 15:57:03.324322 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:03.324302    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:57:04.017605 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:04.017567    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" event={"ID":"65f48e48-a6b1-449e-b63b-44d9789304cf","Type":"ContainerStarted","Data":"15a815731387979106d134581bd4e8c69f7eb45e4a68e21d4754080fdf3723e1"}
Nov 03 15:57:05.023680 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:05.023642    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" event={"ID":"65f48e48-a6b1-449e-b63b-44d9789304cf","Type":"ContainerStarted","Data":"86773fa59428582287433357414ef871004de529647b43747a12686158d7e153"}
Nov 03 15:57:05.024180 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:05.023762    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:05.148286 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:05.148242    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" podStartSLOduration=3.2264922990000002 podStartE2EDuration="4.148228634s" totalImagesPullingTime="921.736335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:57:05.147731953 +0000 UTC m=+2940.072549869" watchObservedRunningTime="2026-11-03 15:57:05.148228634 +0000 UTC m=+2940.073046546"
Nov 03 15:57:10.347546 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:10.347497    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:57:11.055946 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:11.055915    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:57:12.010614 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:12.010585    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:57:12.011134 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:12.010663    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:57:19.100922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:19.100825    4955 generic.go:356] "Generic (PLEG): container finished" podID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerID="b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4" exitCode=0
Nov 03 15:57:19.100922 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:19.100880    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" event={"ID":"3804ce19-0c5d-4568-9a41-df8d8765b172","Type":"ContainerDied","Data":"b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"}
Nov 03 15:57:19.101549 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:19.101379    4955 scope.go:122] "RemoveContainer" containerID="b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"
Nov 03 15:57:19.102675 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:19.102640    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:57:19.102767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:19.102710    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:57:19.102767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:19.102745    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 15:57:20.107839 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:20.107803    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" event={"ID":"3804ce19-0c5d-4568-9a41-df8d8765b172","Type":"ContainerStarted","Data":"a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17"}
Nov 03 15:57:20.916899 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:20.916866    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:57:20.917772 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:20.917748    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 15:57:20.917826 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:20.917799    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 15:57:21.917583 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:21.917545    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 15:57:21.918046 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:21.917637    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 15:57:22.917275 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:22.917238    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 15:57:22.917480 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:22.917333    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 15:57:23.917033 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:23.916990    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 15:57:23.917422 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:23.917052    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 15:57:23.968242 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:23.968203    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:57:24.916730 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:24.916690    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Nov 03 15:57:24.916925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:24.916758    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Nov 03 15:57:25.920819 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:25.920785    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:57:25.922024 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:25.922003    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:57:27.937230 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:57:27.937191    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:57:29.995558 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:29.995524    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:57:55.302698 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:57:55.302646    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:57:57.938606 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:57:57.938570    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:58:05.873329 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:05.873295    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:58:05.889459 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:05.889431    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:58:05.892340 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:05.892319    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 15:58:05.899950 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:05.899928    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 15:58:24.521174 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:24.521137    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:58:27.938695 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:58:27.938655    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:58:33.376527 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:33.376487    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"]
Nov 03 15:58:33.482121 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:33.482077    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"]
Nov 03 15:58:38.377166 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:38.377083    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerName="payload-processing" containerID="cri-o://fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6" gracePeriod=25
Nov 03 15:58:38.482537 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:38.482497    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerName="payload-pre-processing" containerID="cri-o://f9f50e59785abd7cc203ea0dd656f3620ce46af9b57edd9962c35752c64822fc" gracePeriod=25
Nov 03 15:58:40.607636 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:40.607600    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"]
Nov 03 15:58:40.608176 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:40.607838    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" podUID="65f48e48-a6b1-449e-b63b-44d9789304cf" containerName="maas-api" containerID="cri-o://86773fa59428582287433357414ef871004de529647b43747a12686158d7e153" gracePeriod=30
Nov 03 15:58:41.029754 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.029704    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" podUID="65f48e48-a6b1-449e-b63b-44d9789304cf" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.141:8443/health\": dial tcp 10.132.0.141:8443: connect: connection refused"
Nov 03 15:58:41.864211 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.864178    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:58:41.885410 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.885382    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca\") pod \"65f48e48-a6b1-449e-b63b-44d9789304cf\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") "
Nov 03 15:58:41.885573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.885477    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\") pod \"65f48e48-a6b1-449e-b63b-44d9789304cf\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") "
Nov 03 15:58:41.885573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.885513    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm\" (UniqueName: \"kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm\") pod \"65f48e48-a6b1-449e-b63b-44d9789304cf\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") "
Nov 03 15:58:41.885573 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.885564    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs\") pod \"65f48e48-a6b1-449e-b63b-44d9789304cf\" (UID: \"65f48e48-a6b1-449e-b63b-44d9789304cf\") "
Nov 03 15:58:41.885767 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.885742    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca" pod "65f48e48-a6b1-449e-b63b-44d9789304cf" (UID: "65f48e48-a6b1-449e-b63b-44d9789304cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:58:41.885887 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.885864    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f48e48-a6b1-449e-b63b-44d9789304cf-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:41.887856 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.887830    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls" pod "65f48e48-a6b1-449e-b63b-44d9789304cf" (UID: "65f48e48-a6b1-449e-b63b-44d9789304cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:58:41.887997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.887872    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm" pod "65f48e48-a6b1-449e-b63b-44d9789304cf" (UID: "65f48e48-a6b1-449e-b63b-44d9789304cf"). InnerVolumeSpecName "kube-api-access-khtpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:58:41.887997 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.887869    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs" pod "65f48e48-a6b1-449e-b63b-44d9789304cf" (UID: "65f48e48-a6b1-449e-b63b-44d9789304cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:58:41.986665 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.986586    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:41.986665 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.986616    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khtpm\" (UniqueName: \"kubernetes.io/projected/65f48e48-a6b1-449e-b63b-44d9789304cf-kube-api-access-khtpm\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:41.986665 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:41.986626    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f48e48-a6b1-449e-b63b-44d9789304cf-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:42.008854 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.008813    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:58:42.009060 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.008895    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:58:42.569105 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.569044    4955 generic.go:356] "Generic (PLEG): container finished" podID="65f48e48-a6b1-449e-b63b-44d9789304cf" containerID="86773fa59428582287433357414ef871004de529647b43747a12686158d7e153" exitCode=0
Nov 03 15:58:42.569589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.569563    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" event={"ID":"65f48e48-a6b1-449e-b63b-44d9789304cf","Type":"ContainerDied","Data":"86773fa59428582287433357414ef871004de529647b43747a12686158d7e153"}
Nov 03 15:58:42.569868 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.569835    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2" event={"ID":"65f48e48-a6b1-449e-b63b-44d9789304cf","Type":"ContainerDied","Data":"15a815731387979106d134581bd4e8c69f7eb45e4a68e21d4754080fdf3723e1"}
Nov 03 15:58:42.570074 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.570043    4955 scope.go:122] "RemoveContainer" containerID="86773fa59428582287433357414ef871004de529647b43747a12686158d7e153"
Nov 03 15:58:42.570574 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.570556    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"
Nov 03 15:58:42.598081 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.597999    4955 scope.go:122] "RemoveContainer" containerID="86773fa59428582287433357414ef871004de529647b43747a12686158d7e153"
Nov 03 15:58:42.604077 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:58:42.600450    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86773fa59428582287433357414ef871004de529647b43747a12686158d7e153\": container with ID starting with 86773fa59428582287433357414ef871004de529647b43747a12686158d7e153 not found: ID does not exist" containerID="86773fa59428582287433357414ef871004de529647b43747a12686158d7e153"
Nov 03 15:58:42.604077 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.600500    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86773fa59428582287433357414ef871004de529647b43747a12686158d7e153"} err="failed to get container status \"86773fa59428582287433357414ef871004de529647b43747a12686158d7e153\": rpc error: code = NotFound desc = could not find container \"86773fa59428582287433357414ef871004de529647b43747a12686158d7e153\": container with ID starting with 86773fa59428582287433357414ef871004de529647b43747a12686158d7e153 not found: ID does not exist"
Nov 03 15:58:42.737379 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.737330    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"]
Nov 03 15:58:42.787446 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:42.787413    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9d854-6658867448-8d5f2"]
Nov 03 15:58:43.204562 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:43.204534    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f48e48-a6b1-449e-b63b-44d9789304cf" path="/var/lib/kubelet/pods/65f48e48-a6b1-449e-b63b-44d9789304cf/volumes"
Nov 03 15:58:45.581884 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:45.581846    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:58:50.177693 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:50.177655    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"]
Nov 03 15:58:50.178153 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:50.177857    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy" containerID="cri-o://a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17" gracePeriod=30
Nov 03 15:58:52.009314 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:52.009267    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:58:52.009878 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:52.009853    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:58:55.432099 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.432070    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:58:55.515278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515192    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515278 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515234    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515531 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515381    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515531 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515440    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515531 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515487    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515536    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515588    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515682 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515661    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515721    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:58:55.515847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515727    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket\") pod \"3804ce19-0c5d-4568-9a41-df8d8765b172\" (UID: \"3804ce19-0c5d-4568-9a41-df8d8765b172\") "
Nov 03 15:58:55.515847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515720    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:58:55.515847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515736    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:58:55.515847 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.515790    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:58:55.516149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.516027    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:58:55.516149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.516123    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.516149 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.516137    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.516289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.516152    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.516289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.516164    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.516289 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.516172    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804ce19-0c5d-4568-9a41-df8d8765b172-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.517696 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.517669    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:58:55.517925 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.517909    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "kube-api-access-6svjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:58:55.518398 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.518376    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:58:55.518472 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.518407    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token" pod "3804ce19-0c5d-4568-9a41-df8d8765b172" (UID: "3804ce19-0c5d-4568-9a41-df8d8765b172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:58:55.617483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.617445    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.617483 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.617485    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.617699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.617498    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804ce19-0c5d-4568-9a41-df8d8765b172-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.617699 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.617510    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svjt\" (UniqueName: \"kubernetes.io/projected/3804ce19-0c5d-4568-9a41-df8d8765b172-kube-api-access-6svjt\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:55.634408 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.634364    4955 generic.go:356] "Generic (PLEG): container finished" podID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerID="a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17" exitCode=0
Nov 03 15:58:55.634610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.634417    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"
Nov 03 15:58:55.634610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.634436    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" event={"ID":"3804ce19-0c5d-4568-9a41-df8d8765b172","Type":"ContainerDied","Data":"a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17"}
Nov 03 15:58:55.634610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.634493    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp" event={"ID":"3804ce19-0c5d-4568-9a41-df8d8765b172","Type":"ContainerDied","Data":"43e2588d1930ae680cefed4f34ae3a5a24065f91243d937409d4dfa798be3fa0"}
Nov 03 15:58:55.634610 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.634516    4955 scope.go:122] "RemoveContainer" containerID="a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17"
Nov 03 15:58:55.646005 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.645959    4955 scope.go:122] "RemoveContainer" containerID="b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"
Nov 03 15:58:55.655589 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.655566    4955 scope.go:122] "RemoveContainer" containerID="a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17"
Nov 03 15:58:55.655914 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:58:55.655883    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17\": container with ID starting with a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17 not found: ID does not exist" containerID="a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17"
Nov 03 15:58:55.656028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.655928    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17"} err="failed to get container status \"a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17\": rpc error: code = NotFound desc = could not find container \"a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17\": container with ID starting with a3314ce18ccac4a31d4a9e8e6f83df4ea7c4a8fbd0068db38a5118e4eb228b17 not found: ID does not exist"
Nov 03 15:58:55.656028 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.655953    4955 scope.go:122] "RemoveContainer" containerID="b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"
Nov 03 15:58:55.656304 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:58:55.656283    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4\": container with ID starting with b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4 not found: ID does not exist" containerID="b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"
Nov 03 15:58:55.656359 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.656309    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4"} err="failed to get container status \"b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4\": rpc error: code = NotFound desc = could not find container \"b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4\": container with ID starting with b25a62e5ac387d72788a53d2ef99ee7a2b5e206a8c5ca778c41dc6b7500d03b4 not found: ID does not exist"
Nov 03 15:58:55.802802 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.802715    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"]
Nov 03 15:58:55.806801 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:55.806776    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9d854-openshift-default-6bf846ff69j2dxp"]
Nov 03 15:58:57.204690 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:57.204653    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" path="/var/lib/kubelet/pods/3804ce19-0c5d-4568-9a41-df8d8765b172/volumes"
Nov 03 15:58:57.938570 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:58:57.938537    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:58:58.634010 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.633934    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:58:58.652062 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.652032    4955 generic.go:356] "Generic (PLEG): container finished" podID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerID="fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6" exitCode=0
Nov 03 15:58:58.652239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.652081    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"
Nov 03 15:58:58.652239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.652125    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" event={"ID":"0e2f23bd-e165-4e32-bd81-2612317fa00a","Type":"ContainerDied","Data":"fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6"}
Nov 03 15:58:58.652239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.652192    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67" event={"ID":"0e2f23bd-e165-4e32-bd81-2612317fa00a","Type":"ContainerDied","Data":"91689a85eb1499c2dc818b010cf00384c66e6dcd73825bc771dfbb5f9a2c7d90"}
Nov 03 15:58:58.652239 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.652225    4955 scope.go:122] "RemoveContainer" containerID="fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6"
Nov 03 15:58:58.654167 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.654144    4955 generic.go:356] "Generic (PLEG): container finished" podID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerID="f9f50e59785abd7cc203ea0dd656f3620ce46af9b57edd9962c35752c64822fc" exitCode=0
Nov 03 15:58:58.654420 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.654327    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" event={"ID":"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302","Type":"ContainerDied","Data":"f9f50e59785abd7cc203ea0dd656f3620ce46af9b57edd9962c35752c64822fc"}
Nov 03 15:58:58.664870 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.664632    4955 scope.go:122] "RemoveContainer" containerID="fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6"
Nov 03 15:58:58.665102 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:58:58.665082    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6\": container with ID starting with fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6 not found: ID does not exist" containerID="fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6"
Nov 03 15:58:58.665186 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.665115    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6"} err="failed to get container status \"fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6\": rpc error: code = NotFound desc = could not find container \"fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6\": container with ID starting with fe53b24f1c8262fa3c6d26dd80d3172ba485dd644c78ab60978f6c5d5ff53dc6 not found: ID does not exist"
Nov 03 15:58:58.730150 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.730125    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:58:58.747098 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.747073    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v\" (UniqueName: \"kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v\") pod \"0e2f23bd-e165-4e32-bd81-2612317fa00a\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") "
Nov 03 15:58:58.747175 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.747153    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume\") pod \"0e2f23bd-e165-4e32-bd81-2612317fa00a\" (UID: \"0e2f23bd-e165-4e32-bd81-2612317fa00a\") "
Nov 03 15:58:58.747517 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.747497    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume" pod "0e2f23bd-e165-4e32-bd81-2612317fa00a" (UID: "0e2f23bd-e165-4e32-bd81-2612317fa00a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:58:58.749430 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.749410    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v" pod "0e2f23bd-e165-4e32-bd81-2612317fa00a" (UID: "0e2f23bd-e165-4e32-bd81-2612317fa00a"). InnerVolumeSpecName "kube-api-access-gz76v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:58:58.848630 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.848591    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume\") pod \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") "
Nov 03 15:58:58.848806 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.848693    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n\" (UniqueName: \"kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n\") pod \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\" (UID: \"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302\") "
Nov 03 15:58:58.848991 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.848947    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume" pod "1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" (UID: "1d82c8c4-2a5d-4c2b-9984-5dc07fff6302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:58:58.849063 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.849044    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz76v\" (UniqueName: \"kubernetes.io/projected/0e2f23bd-e165-4e32-bd81-2612317fa00a-kube-api-access-gz76v\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:58.849102 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.849071    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2f23bd-e165-4e32-bd81-2612317fa00a-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:58.851000 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.850980    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n" pod "1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" (UID: "1d82c8c4-2a5d-4c2b-9984-5dc07fff6302"). InnerVolumeSpecName "kube-api-access-wsb7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:58:58.950366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.950326    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsb7n\" (UniqueName: \"kubernetes.io/projected/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-kube-api-access-wsb7n\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:58.950366 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:58.950357    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 15:58:59.074411 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.074373    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"]
Nov 03 15:58:59.079083 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.079054    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9d854-754ffcdcbtbw67"]
Nov 03 15:58:59.205030 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.204937    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" path="/var/lib/kubelet/pods/0e2f23bd-e165-4e32-bd81-2612317fa00a/volumes"
Nov 03 15:58:59.660414 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.660378    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"
Nov 03 15:58:59.660911 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.660412    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh" event={"ID":"1d82c8c4-2a5d-4c2b-9984-5dc07fff6302","Type":"ContainerDied","Data":"c5c0c06b572b9d46c9bc255326ead32594d52c50ed6c3ffb59860c26e336a0d8"}
Nov 03 15:58:59.660911 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.660480    4955 scope.go:122] "RemoveContainer" containerID="f9f50e59785abd7cc203ea0dd656f3620ce46af9b57edd9962c35752c64822fc"
Nov 03 15:58:59.827719 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.827678    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"]
Nov 03 15:58:59.831735 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:58:59.831707    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9d854-67cd6v62fh"]
Nov 03 15:59:01.204421 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:59:01.204386    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" path="/var/lib/kubelet/pods/1d82c8c4-2a5d-4c2b-9984-5dc07fff6302/volumes"
Nov 03 15:59:06.870047 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:59:06.869923    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:59:17.461070 ip-10-0-134-23 kubenswrapper[4955]: I1103 15:59:17.461031    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 15:59:27.938718 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:59:27.938678    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 15:59:57.937845 ip-10-0-134-23 kubenswrapper[4955]: E1103 15:59:57.937800    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:00:27.945433 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:00:27.945368    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:00:52.780814 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:00:52.780776    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:00:55.628118 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:00:55.628076    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:00:55.628781 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:00:55.628750    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:00:55.735796 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:00:55.735758    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:00:55.736143 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:00:55.736112    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:00:57.938519 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:00:57.938479    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:01:10.054066 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:10.054022    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:01:19.495098 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:19.495062    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:01:27.938497 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:01:27.938454    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:01:39.177779 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.177747    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:01:39.289921 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.289885    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"]
Nov 03 16:01:39.291182 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291154    4955 cpu_manager.go:414] "containerMap: removing container" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerName="payload-pre-processing"
Nov 03 16:01:39.291182 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291182    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerName="payload-pre-processing"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291207    4955 cpu_manager.go:414] "containerMap: removing container" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerName="payload-processing"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291215    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerName="payload-processing"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291231    4955 cpu_manager.go:414] "containerMap: removing container" podUID="65f48e48-a6b1-449e-b63b-44d9789304cf" containerName="maas-api"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291239    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f48e48-a6b1-449e-b63b-44d9789304cf" containerName="maas-api"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291267    4955 cpu_manager.go:414] "containerMap: removing container" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291274    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291290    4955 cpu_manager.go:414] "containerMap: removing container" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291297    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291391    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy"
Nov 03 16:01:39.291403 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291400    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="3804ce19-0c5d-4568-9a41-df8d8765b172" containerName="istio-proxy"
Nov 03 16:01:39.291935 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291414    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d82c8c4-2a5d-4c2b-9984-5dc07fff6302" containerName="payload-pre-processing"
Nov 03 16:01:39.291935 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291425    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2f23bd-e165-4e32-bd81-2612317fa00a" containerName="payload-processing"
Nov 03 16:01:39.291935 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.291433    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f48e48-a6b1-449e-b63b-44d9789304cf" containerName="maas-api"
Nov 03 16:01:39.295027 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.295003    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.364288 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.364250    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-15832f-openshift-default-dockercfg-vjnjv\""
Nov 03 16:01:39.401322 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.401289    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"]
Nov 03 16:01:39.449324 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449236    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449324 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449275    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449324 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449295    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449565 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449453    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5vp\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449565 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449490    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449565 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449508    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449666 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449615    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449666 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449659    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.449731 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.449694    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.550849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.550814    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.550849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.550855    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.550884    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.550913    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.550933    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551034    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551122    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5vp\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551149    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551167 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551168    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551524 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551443    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551666 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551645    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551805 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551762    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551805 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551794    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.551953 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.551872    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.553624 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.553602    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.553756 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.553737    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.617883 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.617845    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5vp\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.634998 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.628475    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token\") pod \"e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") " pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:39.637687 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:39.637662    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:40.097760 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.097728    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"]
Nov 03 16:01:40.100345 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:01:40.100316    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97925091_0afb_4299_8639_61407a1c359d.slice/crio-36d0571a0d2c0f329cf35a9ead5b7612e0b15bd583a870e38e58cd507dd8d686 WatchSource:0}: Error finding container 36d0571a0d2c0f329cf35a9ead5b7612e0b15bd583a870e38e58cd507dd8d686: Status 404 returned error can't find the container with id 36d0571a0d2c0f329cf35a9ead5b7612e0b15bd583a870e38e58cd507dd8d686
Nov 03 16:01:40.102596 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.102564    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:01:40.102673 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.102637    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:01:40.102673 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.102667    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:01:40.520193 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.520158    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" event={"ID":"97925091-0afb-4299-8639-61407a1c359d","Type":"ContainerStarted","Data":"e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"}
Nov 03 16:01:40.520619 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.520212    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" event={"ID":"97925091-0afb-4299-8639-61407a1c359d","Type":"ContainerStarted","Data":"36d0571a0d2c0f329cf35a9ead5b7612e0b15bd583a870e38e58cd507dd8d686"}
Nov 03 16:01:40.639019 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.638984    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:40.645422 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:40.645366    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podStartSLOduration=1.645351749 podStartE2EDuration="1.645351749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:40.643261697 +0000 UTC m=+3215.568079614" watchObservedRunningTime="2026-11-03 16:01:40.645351749 +0000 UTC m=+3215.570169661"
Nov 03 16:01:41.643834 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:41.643804    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:42.530308 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:42.530274    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:42.531355 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:42.531336    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:01:47.380140 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.380100    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"]
Nov 03 16:01:47.383784 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.383765    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.437783 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.437740    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-15832f\""
Nov 03 16:01:47.492132 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.492098    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cd1fc6d6-dockercfg-txl85\""
Nov 03 16:01:47.494998 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.494948    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"]
Nov 03 16:01:47.539610 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.539562    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2pz\" (UniqueName: \"kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz\") pod \"payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.539803 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.539635    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.544550 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.544510    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"]
Nov 03 16:01:47.548656 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.548633    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.600932 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.600891    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"]
Nov 03 16:01:47.641061 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.641037    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.641240 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.641081    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlq9\" (UniqueName: \"kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9\") pod \"payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.641240 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.641108    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2pz\" (UniqueName: \"kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz\") pod \"payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.641240 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.641228    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.641946 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.641921    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.703639 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.703608    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2pz\" (UniqueName: \"kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz\") pod \"payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:47.742766 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.742728    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlq9\" (UniqueName: \"kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9\") pod \"payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.742935 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.742907    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.743630 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.743608    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.805095 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.805063    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlq9\" (UniqueName: \"kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9\") pod \"payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.862836 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.862787    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:47.997990 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:47.997928    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:48.240990 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.240943    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"]
Nov 03 16:01:48.242820 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:01:48.242792    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7642e8c_0fce_4b53_921e_f9fcf111ef63.slice/crio-03494f1d176146ee64f1a18176c27d75502a35029ff8150fef1ff2340986ed26 WatchSource:0}: Error finding container 03494f1d176146ee64f1a18176c27d75502a35029ff8150fef1ff2340986ed26: Status 404 returned error can't find the container with id 03494f1d176146ee64f1a18176c27d75502a35029ff8150fef1ff2340986ed26
Nov 03 16:01:48.369645 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.369623    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"]
Nov 03 16:01:48.371850 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:01:48.371821    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57c227f_d26e_4764_ad7b_35f22cb3915a.slice/crio-07ede52a5403f53758fc360ee018432270cf42c850234ea88752a55626620153 WatchSource:0}: Error finding container 07ede52a5403f53758fc360ee018432270cf42c850234ea88752a55626620153: Status 404 returned error can't find the container with id 07ede52a5403f53758fc360ee018432270cf42c850234ea88752a55626620153
Nov 03 16:01:48.560267 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.560175    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" event={"ID":"a7642e8c-0fce-4b53-921e-f9fcf111ef63","Type":"ContainerStarted","Data":"750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b"}
Nov 03 16:01:48.560267 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.560227    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" event={"ID":"a7642e8c-0fce-4b53-921e-f9fcf111ef63","Type":"ContainerStarted","Data":"03494f1d176146ee64f1a18176c27d75502a35029ff8150fef1ff2340986ed26"}
Nov 03 16:01:48.560267 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.560254    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:48.561858 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.561834    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" event={"ID":"e57c227f-d26e-4764-ad7b-35f22cb3915a","Type":"ContainerStarted","Data":"1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038"}
Nov 03 16:01:48.562001 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.561875    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" event={"ID":"e57c227f-d26e-4764-ad7b-35f22cb3915a","Type":"ContainerStarted","Data":"07ede52a5403f53758fc360ee018432270cf42c850234ea88752a55626620153"}
Nov 03 16:01:48.562001 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.561919    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:01:48.678540 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.678487    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" podStartSLOduration=1.6784738780000001 podStartE2EDuration="1.678473878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:48.67685403 +0000 UTC m=+3223.601671957" watchObservedRunningTime="2026-11-03 16:01:48.678473878 +0000 UTC m=+3223.603291790"
Nov 03 16:01:48.806552 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:48.806488    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" podStartSLOduration=1.806465821 podStartE2EDuration="1.806465821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:48.799345581 +0000 UTC m=+3223.724163502" watchObservedRunningTime="2026-11-03 16:01:48.806465821 +0000 UTC m=+3223.731283738"
Nov 03 16:01:49.922462 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:49.922424    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"]
Nov 03 16:01:49.927079 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:49.927060    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:49.980887 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:49.980848    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-15832f-serving-cert\""
Nov 03 16:01:50.033745 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.033710    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"]
Nov 03 16:01:50.066334 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.066302    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.066500 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.066347    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.066552 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.066512    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.066552 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.066547    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlg2\" (UniqueName: \"kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.168825 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.168776    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.169068 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.168854    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlg2\" (UniqueName: \"kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.169068 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.168908    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.169068 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.168949    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.169930 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.169898    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.171893 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:01:50.171870    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-serving-cert: secret "maas-api-e2e-worker-single-main-15832f-serving-cert" not found
Nov 03 16:01:50.172129 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:01:50.172115    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls podName:6c12dbb4-06ba-47d1-80ff-257eaaba5f40 nodeName:}" failed. No retries permitted until 2026-11-03 16:01:50.67209212 +0000 UTC m=+3225.596910025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls") pod "maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" (UID: "6c12dbb4-06ba-47d1-80ff-257eaaba5f40") : secret "maas-api-e2e-worker-single-main-15832f-serving-cert" not found
Nov 03 16:01:50.178688 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.178624    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.234412 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.234381    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlg2\" (UniqueName: \"kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.682837 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.682790    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.685456 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.685431    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:50.845367 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:50.845327    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:51.228011 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:51.227982    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"]
Nov 03 16:01:51.229862 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:01:51.229833    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c12dbb4_06ba_47d1_80ff_257eaaba5f40.slice/crio-80d2637d64554c447fa341ff2a0ada1a19f4b03f41df4ca635005f9eb9c183f2 WatchSource:0}: Error finding container 80d2637d64554c447fa341ff2a0ada1a19f4b03f41df4ca635005f9eb9c183f2: Status 404 returned error can't find the container with id 80d2637d64554c447fa341ff2a0ada1a19f4b03f41df4ca635005f9eb9c183f2
Nov 03 16:01:51.577208 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:51.577126    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" event={"ID":"6c12dbb4-06ba-47d1-80ff-257eaaba5f40","Type":"ContainerStarted","Data":"80d2637d64554c447fa341ff2a0ada1a19f4b03f41df4ca635005f9eb9c183f2"}
Nov 03 16:01:52.583467 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:52.583431    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" event={"ID":"6c12dbb4-06ba-47d1-80ff-257eaaba5f40","Type":"ContainerStarted","Data":"7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d"}
Nov 03 16:01:52.583904 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:52.583569    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:52.704487 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:52.704432    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" podStartSLOduration=2.800934897 podStartE2EDuration="3.704417666s" totalImagesPullingTime="903.482769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:01:52.703372997 +0000 UTC m=+3227.628190912" watchObservedRunningTime="2026-11-03 16:01:52.704417666 +0000 UTC m=+3227.629235578"
Nov 03 16:01:57.938953 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:01:57.938912    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:01:58.361577 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:58.361485    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:01:58.595020 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:58.594989    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:01:59.569112 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:59.569084    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:01:59.569608 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:01:59.569153    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:02:08.682263 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:08.682227    4955 generic.go:356] "Generic (PLEG): container finished" podID="97925091-0afb-4299-8639-61407a1c359d" containerID="e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c" exitCode=0
Nov 03 16:02:08.682703 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:08.682294    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" event={"ID":"97925091-0afb-4299-8639-61407a1c359d","Type":"ContainerDied","Data":"e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"}
Nov 03 16:02:08.682703 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:08.682669    4955 scope.go:122] "RemoveContainer" containerID="e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"
Nov 03 16:02:08.683860 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:08.683828    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:02:08.684009 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:08.683918    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:02:08.684009 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:08.683995    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:02:09.692892 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:09.692853    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" event={"ID":"97925091-0afb-4299-8639-61407a1c359d","Type":"ContainerStarted","Data":"04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd"}
Nov 03 16:02:10.640339 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:10.640306    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:02:10.653995 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:10.651119    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 16:02:10.653995 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:10.651205    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 16:02:11.639710 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:11.639676    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 16:02:11.654234 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:11.654106    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 16:02:12.537090 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:12.537054    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:02:12.639325 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:12.639288    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 16:02:12.639626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:12.639593    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 16:02:13.651993 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:13.651092    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 16:02:13.651993 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:13.651159    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 16:02:14.638658 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:14.638625    4955 patch_prober.go:29] interesting pod/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 03 16:02:14.638842 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:14.638682    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 03 16:02:15.642622 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:15.642591    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:02:15.643545 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:15.643525    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:02:17.217539 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:17.217506    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:02:27.941435 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:02:27.941389    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:02:44.580285 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:02:44.580187    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:02:57.941144 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:02:57.941098    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:03:05.917394 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:05.917360    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 16:03:05.926185 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:05.926163    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 16:03:05.927819 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:05.927798    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 16:03:05.935437 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:05.935421    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 16:03:10.592366 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:10.592327    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:03:20.628154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:20.628110    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"]
Nov 03 16:03:20.734644 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:20.734609    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"]
Nov 03 16:03:25.628593 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:25.628546    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerName="payload-processing" containerID="cri-o://750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b" gracePeriod=25
Nov 03 16:03:25.735416 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:25.735363    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerName="payload-pre-processing" containerID="cri-o://1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038" gracePeriod=25
Nov 03 16:03:27.938087 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:27.938049    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:03:28.219962 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:28.219878    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"]
Nov 03 16:03:28.220156 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:28.220098    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" podUID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" containerName="maas-api" containerID="cri-o://7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d" gracePeriod=30
Nov 03 16:03:28.589462 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:28.589377    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" podUID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.145:8443/health\": dial tcp 10.132.0.145:8443: connect: connection refused"
Nov 03 16:03:29.479293 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.479259    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:03:29.510935 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.510900    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2\" (UniqueName: \"kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2\") pod \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") "
Nov 03 16:03:29.510935 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.510942    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\") pod \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") "
Nov 03 16:03:29.511183 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.510997    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca\") pod \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") "
Nov 03 16:03:29.511183 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.511061    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs\") pod \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\" (UID: \"6c12dbb4-06ba-47d1-80ff-257eaaba5f40\") "
Nov 03 16:03:29.511423 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.511393    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca" pod "6c12dbb4-06ba-47d1-80ff-257eaaba5f40" (UID: "6c12dbb4-06ba-47d1-80ff-257eaaba5f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:29.513350 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.513316    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs" pod "6c12dbb4-06ba-47d1-80ff-257eaaba5f40" (UID: "6c12dbb4-06ba-47d1-80ff-257eaaba5f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:29.513438 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.513389    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls" pod "6c12dbb4-06ba-47d1-80ff-257eaaba5f40" (UID: "6c12dbb4-06ba-47d1-80ff-257eaaba5f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:03:29.513438 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.513419    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2" pod "6c12dbb4-06ba-47d1-80ff-257eaaba5f40" (UID: "6c12dbb4-06ba-47d1-80ff-257eaaba5f40"). InnerVolumeSpecName "kube-api-access-4tlg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:29.567847 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.567760    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:03:29.568020 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.567857    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:03:29.612614 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.612573    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:29.612614 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.612606    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tlg2\" (UniqueName: \"kubernetes.io/projected/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-kube-api-access-4tlg2\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:29.612614 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.612621    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:29.612859 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:29.612632    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c12dbb4-06ba-47d1-80ff-257eaaba5f40-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:30.096367 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.096331    4955 generic.go:356] "Generic (PLEG): container finished" podID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" containerID="7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d" exitCode=0
Nov 03 16:03:30.096580 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.096378    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"
Nov 03 16:03:30.096580 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.096422    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" event={"ID":"6c12dbb4-06ba-47d1-80ff-257eaaba5f40","Type":"ContainerDied","Data":"7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d"}
Nov 03 16:03:30.096580 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.096481    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f" event={"ID":"6c12dbb4-06ba-47d1-80ff-257eaaba5f40","Type":"ContainerDied","Data":"80d2637d64554c447fa341ff2a0ada1a19f4b03f41df4ca635005f9eb9c183f2"}
Nov 03 16:03:30.096580 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.096509    4955 scope.go:122] "RemoveContainer" containerID="7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d"
Nov 03 16:03:30.105839 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.105816    4955 scope.go:122] "RemoveContainer" containerID="7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d"
Nov 03 16:03:30.106140 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:30.106118    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d\": container with ID starting with 7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d not found: ID does not exist" containerID="7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d"
Nov 03 16:03:30.106219 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.106152    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d"} err="failed to get container status \"7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d\": rpc error: code = NotFound desc = could not find container \"7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d\": container with ID starting with 7c5afc9d20421959db444fab6d0ae1f49269286a40a565e379e1c6457e64fb7d not found: ID does not exist"
Nov 03 16:03:30.262456 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.262421    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"]
Nov 03 16:03:30.312798 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:30.312769    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-15832f-57f4fdfd8f-wp67f"]
Nov 03 16:03:31.206280 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:31.206247    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" path="/var/lib/kubelet/pods/6c12dbb4-06ba-47d1-80ff-257eaaba5f40/volumes"
Nov 03 16:03:35.094347 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:35.094314    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:03:37.335498 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:37.335461    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"]
Nov 03 16:03:37.335900 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:37.335684    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy" containerID="cri-o://04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd" gracePeriod=30
Nov 03 16:03:39.568086 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:39.568048    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:03:39.568476 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:39.568095    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:03:42.589262 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.589234    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:03:42.632658 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632577    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632658 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632644    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632847 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632727    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632847 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632771    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632965 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632850    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632965 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632880    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632965 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632880    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:42.632965 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632909    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.632965 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632942    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.633234 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.632990    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp\") pod \"97925091-0afb-4299-8639-61407a1c359d\" (UID: \"97925091-0afb-4299-8639-61407a1c359d\") "
Nov 03 16:03:42.633234 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.633013    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:42.633341 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.633304    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.633341 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.633321    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.633668 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.633447    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:42.633668 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.633465    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:42.634069 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.634043    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:42.635317 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.635284    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:03:42.635317 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.635310    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:03:42.635766 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.635746    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:42.635851 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.635781    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp" pod "97925091-0afb-4299-8639-61407a1c359d" (UID: "97925091-0afb-4299-8639-61407a1c359d"). InnerVolumeSpecName "kube-api-access-tr5vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:42.734587 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734555    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.734587 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734581    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97925091-0afb-4299-8639-61407a1c359d-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.734587 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734590    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.734808 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734599    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.734808 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734608    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5vp\" (UniqueName: \"kubernetes.io/projected/97925091-0afb-4299-8639-61407a1c359d-kube-api-access-tr5vp\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.734808 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734617    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97925091-0afb-4299-8639-61407a1c359d-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:42.734808 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:42.734626    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97925091-0afb-4299-8639-61407a1c359d-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:43.164344 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.164297    4955 generic.go:356] "Generic (PLEG): container finished" podID="97925091-0afb-4299-8639-61407a1c359d" containerID="04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd" exitCode=0
Nov 03 16:03:43.164564 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.164363    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"
Nov 03 16:03:43.164564 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.164369    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" event={"ID":"97925091-0afb-4299-8639-61407a1c359d","Type":"ContainerDied","Data":"04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd"}
Nov 03 16:03:43.164564 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.164489    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms" event={"ID":"97925091-0afb-4299-8639-61407a1c359d","Type":"ContainerDied","Data":"36d0571a0d2c0f329cf35a9ead5b7612e0b15bd583a870e38e58cd507dd8d686"}
Nov 03 16:03:43.164564 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.164510    4955 scope.go:122] "RemoveContainer" containerID="04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd"
Nov 03 16:03:43.174066 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.174045    4955 scope.go:122] "RemoveContainer" containerID="e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"
Nov 03 16:03:43.182802 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.182780    4955 scope.go:122] "RemoveContainer" containerID="04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd"
Nov 03 16:03:43.183094 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:43.183073    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd\": container with ID starting with 04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd not found: ID does not exist" containerID="04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd"
Nov 03 16:03:43.183193 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.183109    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd"} err="failed to get container status \"04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd\": rpc error: code = NotFound desc = could not find container \"04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd\": container with ID starting with 04c3ed299146b8ca07d48db6a5a5ade7694e6504f8744b3f9247a306a0d0d3bd not found: ID does not exist"
Nov 03 16:03:43.183193 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.183143    4955 scope.go:122] "RemoveContainer" containerID="e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"
Nov 03 16:03:43.183396 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:43.183378    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c\": container with ID starting with e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c not found: ID does not exist" containerID="e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"
Nov 03 16:03:43.183438 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.183406    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c"} err="failed to get container status \"e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c\": rpc error: code = NotFound desc = could not find container \"e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c\": container with ID starting with e7f8caa32c6ec1fe68d2e27bb03028da5171fdec460dbc3537a2b5bf8e41553c not found: ID does not exist"
Nov 03 16:03:43.332633 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.332597    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"]
Nov 03 16:03:43.337682 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:43.337655    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-15832f-openshift-default-59c4ffd69-lsrms"]
Nov 03 16:03:45.204446 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.204410    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97925091-0afb-4299-8639-61407a1c359d" path="/var/lib/kubelet/pods/97925091-0afb-4299-8639-61407a1c359d/volumes"
Nov 03 16:03:45.889251 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.889187    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:03:45.968323 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.968298    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume\") pod \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") "
Nov 03 16:03:45.968424 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.968355    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9\" (UniqueName: \"kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9\") pod \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\" (UID: \"a7642e8c-0fce-4b53-921e-f9fcf111ef63\") "
Nov 03 16:03:45.968626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.968606    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume" pod "a7642e8c-0fce-4b53-921e-f9fcf111ef63" (UID: "a7642e8c-0fce-4b53-921e-f9fcf111ef63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:45.968727 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.968712    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7642e8c-0fce-4b53-921e-f9fcf111ef63-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:45.970682 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.970657    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9" pod "a7642e8c-0fce-4b53-921e-f9fcf111ef63" (UID: "a7642e8c-0fce-4b53-921e-f9fcf111ef63"). InnerVolumeSpecName "kube-api-access-mdlq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:45.979199 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:45.979179    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:03:46.069629 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.069594    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz\" (UniqueName: \"kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz\") pod \"e57c227f-d26e-4764-ad7b-35f22cb3915a\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") "
Nov 03 16:03:46.069800 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.069638    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume\") pod \"e57c227f-d26e-4764-ad7b-35f22cb3915a\" (UID: \"e57c227f-d26e-4764-ad7b-35f22cb3915a\") "
Nov 03 16:03:46.069867 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.069854    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlq9\" (UniqueName: \"kubernetes.io/projected/a7642e8c-0fce-4b53-921e-f9fcf111ef63-kube-api-access-mdlq9\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:46.070112 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.070080    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume" pod "e57c227f-d26e-4764-ad7b-35f22cb3915a" (UID: "e57c227f-d26e-4764-ad7b-35f22cb3915a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:03:46.071914 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.071897    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz" pod "e57c227f-d26e-4764-ad7b-35f22cb3915a" (UID: "e57c227f-d26e-4764-ad7b-35f22cb3915a"). InnerVolumeSpecName "kube-api-access-nq2pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:03:46.171029 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.170994    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e57c227f-d26e-4764-ad7b-35f22cb3915a-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:46.171029 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.171025    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2pz\" (UniqueName: \"kubernetes.io/projected/e57c227f-d26e-4764-ad7b-35f22cb3915a-kube-api-access-nq2pz\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:03:46.182607 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.182578    4955 generic.go:356] "Generic (PLEG): container finished" podID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerID="750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b" exitCode=0
Nov 03 16:03:46.182754 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.182618    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"
Nov 03 16:03:46.182754 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.182652    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" event={"ID":"a7642e8c-0fce-4b53-921e-f9fcf111ef63","Type":"ContainerDied","Data":"750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b"}
Nov 03 16:03:46.182754 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.182700    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c" event={"ID":"a7642e8c-0fce-4b53-921e-f9fcf111ef63","Type":"ContainerDied","Data":"03494f1d176146ee64f1a18176c27d75502a35029ff8150fef1ff2340986ed26"}
Nov 03 16:03:46.182754 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.182732    4955 scope.go:122] "RemoveContainer" containerID="750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b"
Nov 03 16:03:46.184016 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.183992    4955 generic.go:356] "Generic (PLEG): container finished" podID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerID="1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038" exitCode=0
Nov 03 16:03:46.184311 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.184026    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"
Nov 03 16:03:46.184311 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.184074    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" event={"ID":"e57c227f-d26e-4764-ad7b-35f22cb3915a","Type":"ContainerDied","Data":"1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038"}
Nov 03 16:03:46.184311 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.184126    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv" event={"ID":"e57c227f-d26e-4764-ad7b-35f22cb3915a","Type":"ContainerDied","Data":"07ede52a5403f53758fc360ee018432270cf42c850234ea88752a55626620153"}
Nov 03 16:03:46.193074 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.193056    4955 scope.go:122] "RemoveContainer" containerID="750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b"
Nov 03 16:03:46.193348 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:46.193327    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b\": container with ID starting with 750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b not found: ID does not exist" containerID="750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b"
Nov 03 16:03:46.193436 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.193354    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b"} err="failed to get container status \"750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b\": rpc error: code = NotFound desc = could not find container \"750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b\": container with ID starting with 750eb6dd41260b6ca866be391a7e356ae81125d78c50d0106295cf6c4e62825b not found: ID does not exist"
Nov 03 16:03:46.193436 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.193370    4955 scope.go:122] "RemoveContainer" containerID="1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038"
Nov 03 16:03:46.201185 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.201167    4955 scope.go:122] "RemoveContainer" containerID="1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038"
Nov 03 16:03:46.201457 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:46.201436    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038\": container with ID starting with 1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038 not found: ID does not exist" containerID="1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038"
Nov 03 16:03:46.201521 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.201464    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038"} err="failed to get container status \"1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038\": rpc error: code = NotFound desc = could not find container \"1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038\": container with ID starting with 1df6be475c95ba4daec76834a0267116e013e502ed2651500ca22259891b7038 not found: ID does not exist"
Nov 03 16:03:46.349279 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.349239    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"]
Nov 03 16:03:46.354735 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.354705    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-15832f-79695nm2tv"]
Nov 03 16:03:46.466195 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.466117    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"]
Nov 03 16:03:46.471147 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:46.471119    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-15832f-6564649cb29b5c"]
Nov 03 16:03:47.203862 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:47.203829    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" path="/var/lib/kubelet/pods/a7642e8c-0fce-4b53-921e-f9fcf111ef63/volumes"
Nov 03 16:03:47.204260 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:47.204246    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" path="/var/lib/kubelet/pods/e57c227f-d26e-4764-ad7b-35f22cb3915a/volumes"
Nov 03 16:03:55.281223 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:03:55.281185    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:03:58.046307 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:03:58.046227    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:04:19.103497 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:04:19.103453    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:04:27.937901 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:04:27.937860    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:04:47.191948 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:04:47.191909    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:04:57.938349 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:04:57.938307    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:05:02.572395 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:02.572352    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:05:17.983924 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:17.983886    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:05:25.901753 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:25.901717    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:05:27.939090 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:05:27.939054    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:05:36.553422 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:36.553378    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:05:57.492069 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.492036    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:05:57.938686 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:05:57.938643    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:05:57.975857 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.975828    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"]
Nov 03 16:05:57.976240 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976226    4955 cpu_manager.go:414] "containerMap: removing container" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerName="payload-processing"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976241    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerName="payload-processing"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976257    4955 cpu_manager.go:414] "containerMap: removing container" podUID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" containerName="maas-api"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976262    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" containerName="maas-api"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976271    4955 cpu_manager.go:414] "containerMap: removing container" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976275    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976282    4955 cpu_manager.go:414] "containerMap: removing container" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerName="payload-pre-processing"
Nov 03 16:05:57.976292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976287    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerName="payload-pre-processing"
Nov 03 16:05:57.976530 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976371    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c12dbb4-06ba-47d1-80ff-257eaaba5f40" containerName="maas-api"
Nov 03 16:05:57.976530 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976380    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy"
Nov 03 16:05:57.976530 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976386    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57c227f-d26e-4764-ad7b-35f22cb3915a" containerName="payload-pre-processing"
Nov 03 16:05:57.976530 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976394    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy"
Nov 03 16:05:57.976530 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.976402    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7642e8c-0fce-4b53-921e-f9fcf111ef63" containerName="payload-processing"
Nov 03 16:05:57.979474 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:57.979457    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.086632 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.086601    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"]
Nov 03 16:05:58.125596 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.125563    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d298ca-33bd-4db0-a4fc-78955b570eff-metrics-certs\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.125783 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.125620    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d298ca-33bd-4db0-a4fc-78955b570eff-maas-api-tls\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.125783 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.125723    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d298ca-33bd-4db0-a4fc-78955b570eff-openshift-service-ca\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.125783 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.125778    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7zz\" (UniqueName: \"kubernetes.io/projected/f2d298ca-33bd-4db0-a4fc-78955b570eff-kube-api-access-4j7zz\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.227108 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.227016    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7zz\" (UniqueName: \"kubernetes.io/projected/f2d298ca-33bd-4db0-a4fc-78955b570eff-kube-api-access-4j7zz\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.227108 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.227090    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d298ca-33bd-4db0-a4fc-78955b570eff-metrics-certs\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.227351 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.227147    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d298ca-33bd-4db0-a4fc-78955b570eff-maas-api-tls\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.227351 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.227206    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d298ca-33bd-4db0-a4fc-78955b570eff-openshift-service-ca\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.227872 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.227840    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d298ca-33bd-4db0-a4fc-78955b570eff-openshift-service-ca\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.229807 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.229780    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d298ca-33bd-4db0-a4fc-78955b570eff-maas-api-tls\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.229917 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.229839    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d298ca-33bd-4db0-a4fc-78955b570eff-metrics-certs\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.290957 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.290918    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7zz\" (UniqueName: \"kubernetes.io/projected/f2d298ca-33bd-4db0-a4fc-78955b570eff-kube-api-access-4j7zz\") pod \"maas-api-f5bf8854d-tgl9d\" (UID: \"f2d298ca-33bd-4db0-a4fc-78955b570eff\") " pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.291754 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.291733    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:05:58.668671 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.668644    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"]
Nov 03 16:05:58.670875 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:05:58.670848    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d298ca_33bd_4db0_a4fc_78955b570eff.slice/crio-73bd65fa9598dd38ab97c035ae1dda11e78945bfb76af7e099f234f666786ffb WatchSource:0}: Error finding container 73bd65fa9598dd38ab97c035ae1dda11e78945bfb76af7e099f234f666786ffb: Status 404 returned error can't find the container with id 73bd65fa9598dd38ab97c035ae1dda11e78945bfb76af7e099f234f666786ffb
Nov 03 16:05:58.672226 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:58.672209    4955 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:05:59.131958 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:05:59.131855    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d" event={"ID":"f2d298ca-33bd-4db0-a4fc-78955b570eff","Type":"ContainerStarted","Data":"73bd65fa9598dd38ab97c035ae1dda11e78945bfb76af7e099f234f666786ffb"}
Nov 03 16:06:00.138591 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:00.138499    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d" event={"ID":"f2d298ca-33bd-4db0-a4fc-78955b570eff","Type":"ContainerStarted","Data":"2a9a3676d08ef136284f00f1e77f551f77633aa5fa140e611df859138a7357cd"}
Nov 03 16:06:00.138981 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:00.138594    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:06:00.261305 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:00.261253    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d" podStartSLOduration=2.263717128 podStartE2EDuration="3.261237664s" totalImagesPullingTime="997.520536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:00.259557904 +0000 UTC m=+3475.184375819" watchObservedRunningTime="2026-11-03 16:06:00.261237664 +0000 UTC m=+3475.186055571"
Nov 03 16:06:05.261081 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.261036    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"]
Nov 03 16:06:05.261739 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.261719    4955 cpu_manager.go:414] "containerMap: removing container" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy"
Nov 03 16:06:05.261794 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.261742    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97925091-0afb-4299-8639-61407a1c359d" containerName="istio-proxy"
Nov 03 16:06:05.265652 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.265615    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.319636 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.319603    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2cca28a2-openshift-default-dockercfg-s4h9c\""
Nov 03 16:06:05.374623 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.374587    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"]
Nov 03 16:06:05.400733 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.400705    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.400931 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.400740    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.400931 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.400773    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.400931 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.400926    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6lll\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.401106 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.400990    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.401106 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.401052    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.401106 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.401091    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.401216 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.401110    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.401216 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.401139    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502526 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502488    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502526 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502534    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502553    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502585    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502613    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502633    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.502789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502760    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503121 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502850    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6lll\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503121 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502927    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503121 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.503043    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503121 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.502929    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503121 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.503113    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503340 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.503206    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.503492 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.503472    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.505693 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.505672    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.505833 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.505820    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.565044 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.564962    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.567560 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.567534    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6lll\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll\") pod \"e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") " pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:05.584241 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:05.584210    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:06.018549 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.018521    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"]
Nov 03 16:06:06.020645 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:06:06.020622    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c63d7e_714f_4a32_8be9_9925d9a66050.slice/crio-60a4ec4ee6181a36922085fcc86660109c702f9c40c8345e28ef9dafb8ed9478 WatchSource:0}: Error finding container 60a4ec4ee6181a36922085fcc86660109c702f9c40c8345e28ef9dafb8ed9478: Status 404 returned error can't find the container with id 60a4ec4ee6181a36922085fcc86660109c702f9c40c8345e28ef9dafb8ed9478
Nov 03 16:06:06.023154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.023122    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:06.023237 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.023190    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:06.023237 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.023222    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:06.149212 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.149183    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f5bf8854d-tgl9d"
Nov 03 16:06:06.168947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.168909    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" event={"ID":"a3c63d7e-714f-4a32-8be9-9925d9a66050","Type":"ContainerStarted","Data":"0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"}
Nov 03 16:06:06.169160 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.168994    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" event={"ID":"a3c63d7e-714f-4a32-8be9-9925d9a66050","Type":"ContainerStarted","Data":"60a4ec4ee6181a36922085fcc86660109c702f9c40c8345e28ef9dafb8ed9478"}
Nov 03 16:06:06.393897 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.393835    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podStartSLOduration=1.393821964 podStartE2EDuration="1.393821964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:06.391863417 +0000 UTC m=+3481.316681344" watchObservedRunningTime="2026-11-03 16:06:06.393821964 +0000 UTC m=+3481.318639876"
Nov 03 16:06:06.584753 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:06.584709    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:07.589761 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:07.589730    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:08.178297 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:08.178263    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:08.179408 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:08.179388    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:12.564019 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.563982    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"]
Nov 03 16:06:12.567659 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.567641    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.586475 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.586435    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8gp\" (UniqueName: \"kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp\") pod \"payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.586633 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.586545    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.621583 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.621555    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2cca28a2\""
Nov 03 16:06:12.672237 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.672205    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2cca28a2-dockercfg-tgwcr\""
Nov 03 16:06:12.673958 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.673930    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"]
Nov 03 16:06:12.687646 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.687610    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8gp\" (UniqueName: \"kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp\") pod \"payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.687784 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.687676    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.688410 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.688389    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.760091 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.760057    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8gp\" (UniqueName: \"kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp\") pod \"payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:12.881138 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:12.881053    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:13.206802 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.206770    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"]
Nov 03 16:06:13.211291 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.211272    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.255795 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.255449    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"]
Nov 03 16:06:13.258550 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:06:13.258516    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb307eb8c_a013_4b33_96c9_c9d925cc394e.slice/crio-e03c959cc4b997427a240fe614dd46109e6031f3f1c263e2f45b0a54d0663a1d WatchSource:0}: Error finding container e03c959cc4b997427a240fe614dd46109e6031f3f1c263e2f45b0a54d0663a1d: Status 404 returned error can't find the container with id e03c959cc4b997427a240fe614dd46109e6031f3f1c263e2f45b0a54d0663a1d
Nov 03 16:06:13.265077 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.265047    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2cca28a2-serving-cert\""
Nov 03 16:06:13.270644 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.270617    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"]
Nov 03 16:06:13.292902 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.292878    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.293056 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.292918    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjwp\" (UniqueName: \"kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.293056 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.292936    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.293056 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.293038    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.393916 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.393875    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjwp\" (UniqueName: \"kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.393916 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.393913    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.394224 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.393940    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.394224 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.394056    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.394224 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:13.394088    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-serving-cert: secret "maas-api-e2e-ait-2cca28a2-serving-cert" not found
Nov 03 16:06:13.394224 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:13.394162    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls podName:6bd3de93-53b6-4281-9bd7-bb26a0a14067 nodeName:}" failed. No retries permitted until 2026-11-03 16:06:13.894141233 +0000 UTC m=+3488.818959138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls") pod "maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" (UID: "6bd3de93-53b6-4281-9bd7-bb26a0a14067") : secret "maas-api-e2e-ait-2cca28a2-serving-cert" not found
Nov 03 16:06:13.394635 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.394608    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.396561 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.396545    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.467505 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.467411    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjwp\" (UniqueName: \"kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.900221 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.900189    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:13.902832 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:13.902808    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\") pod \"maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:14.124227 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.124165    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:14.209373 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.209326    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc" event={"ID":"b307eb8c-a013-4b33-96c9-c9d925cc394e","Type":"ContainerStarted","Data":"06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05"}
Nov 03 16:06:14.209574 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.209443    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc" event={"ID":"b307eb8c-a013-4b33-96c9-c9d925cc394e","Type":"ContainerStarted","Data":"e03c959cc4b997427a240fe614dd46109e6031f3f1c263e2f45b0a54d0663a1d"}
Nov 03 16:06:14.209728 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.209711    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:14.330826 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.330771    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc" podStartSLOduration=2.330755629 podStartE2EDuration="2.330755629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:14.328723917 +0000 UTC m=+3489.253541830" watchObservedRunningTime="2026-11-03 16:06:14.330755629 +0000 UTC m=+3489.255573542"
Nov 03 16:06:14.549749 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.549671    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"]
Nov 03 16:06:14.553146 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:06:14.553116    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd3de93_53b6_4281_9bd7_bb26a0a14067.slice/crio-cc2ae7a966a55109be8614544ce50f4094ee2cb966a0cb922dea74fa95e2d5a9 WatchSource:0}: Error finding container cc2ae7a966a55109be8614544ce50f4094ee2cb966a0cb922dea74fa95e2d5a9: Status 404 returned error can't find the container with id cc2ae7a966a55109be8614544ce50f4094ee2cb966a0cb922dea74fa95e2d5a9
Nov 03 16:06:14.597824 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.597779    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"]
Nov 03 16:06:14.598213 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:14.598155    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk" podUID="b728134d-dd38-443e-8c8f-6057caa22bac" containerName="maas-api" containerID="cri-o://7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc" gracePeriod=30
Nov 03 16:06:15.215102 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.215062    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" event={"ID":"6bd3de93-53b6-4281-9bd7-bb26a0a14067","Type":"ContainerStarted","Data":"cc2ae7a966a55109be8614544ce50f4094ee2cb966a0cb922dea74fa95e2d5a9"}
Nov 03 16:06:15.863250 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.863222    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 16:06:15.922926 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.922842    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls\") pod \"b728134d-dd38-443e-8c8f-6057caa22bac\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") "
Nov 03 16:06:15.922926 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.922881    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5\" (UniqueName: \"kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5\") pod \"b728134d-dd38-443e-8c8f-6057caa22bac\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") "
Nov 03 16:06:15.923138 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.922993    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs\") pod \"b728134d-dd38-443e-8c8f-6057caa22bac\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") "
Nov 03 16:06:15.923138 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.923063    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca\") pod \"b728134d-dd38-443e-8c8f-6057caa22bac\" (UID: \"b728134d-dd38-443e-8c8f-6057caa22bac\") "
Nov 03 16:06:15.923445 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.923417    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca" pod "b728134d-dd38-443e-8c8f-6057caa22bac" (UID: "b728134d-dd38-443e-8c8f-6057caa22bac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:06:15.925300 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.925275    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs" pod "b728134d-dd38-443e-8c8f-6057caa22bac" (UID: "b728134d-dd38-443e-8c8f-6057caa22bac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:06:15.925300 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.925293    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls" pod "b728134d-dd38-443e-8c8f-6057caa22bac" (UID: "b728134d-dd38-443e-8c8f-6057caa22bac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:06:15.925440 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:15.925308    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5" pod "b728134d-dd38-443e-8c8f-6057caa22bac" (UID: "b728134d-dd38-443e-8c8f-6057caa22bac"). InnerVolumeSpecName "kube-api-access-nphp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:06:16.024124 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.024091    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:16.024124 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.024123    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphp5\" (UniqueName: \"kubernetes.io/projected/b728134d-dd38-443e-8c8f-6057caa22bac-kube-api-access-nphp5\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:16.024124 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.024133    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b728134d-dd38-443e-8c8f-6057caa22bac-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:16.024388 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.024142    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b728134d-dd38-443e-8c8f-6057caa22bac-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:16.221802 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.221710    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" event={"ID":"6bd3de93-53b6-4281-9bd7-bb26a0a14067","Type":"ContainerStarted","Data":"5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2"}
Nov 03 16:06:16.222285 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.221880    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:16.223111 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.223090    4955 generic.go:356] "Generic (PLEG): container finished" podID="b728134d-dd38-443e-8c8f-6057caa22bac" containerID="7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc" exitCode=0
Nov 03 16:06:16.223202 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.223126    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"
Nov 03 16:06:16.223202 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.223135    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk" event={"ID":"b728134d-dd38-443e-8c8f-6057caa22bac","Type":"ContainerDied","Data":"7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc"}
Nov 03 16:06:16.223202 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.223168    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk" event={"ID":"b728134d-dd38-443e-8c8f-6057caa22bac","Type":"ContainerDied","Data":"48d4e0fc7622ea8645282cf4b9a3459dc0ba37e4d2a49141db78cc3bcdeb0412"}
Nov 03 16:06:16.223202 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.223189    4955 scope.go:122] "RemoveContainer" containerID="7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc"
Nov 03 16:06:16.232728 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.232704    4955 scope.go:122] "RemoveContainer" containerID="7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc"
Nov 03 16:06:16.233032 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:16.233012    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc\": container with ID starting with 7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc not found: ID does not exist" containerID="7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc"
Nov 03 16:06:16.233095 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.233041    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc"} err="failed to get container status \"7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc\": rpc error: code = NotFound desc = could not find container \"7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc\": container with ID starting with 7c2ebc3f64e838c2848ed3ef790bb4be9f59f7b5c87ddaf0d82b63d9feac07fc not found: ID does not exist"
Nov 03 16:06:16.341418 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.341370    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" podStartSLOduration=2.518425474 podStartE2EDuration="3.341357639s" totalImagesPullingTime="822.932165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:16.34122241 +0000 UTC m=+3491.266040323" watchObservedRunningTime="2026-11-03 16:06:16.341357639 +0000 UTC m=+3491.266175552"
Nov 03 16:06:16.464088 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.464022    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"]
Nov 03 16:06:16.467160 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:16.467103    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c798699b4-jwtdk"]
Nov 03 16:06:17.204864 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:17.204829    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b728134d-dd38-443e-8c8f-6057caa22bac" path="/var/lib/kubelet/pods/b728134d-dd38-443e-8c8f-6057caa22bac/volumes"
Nov 03 16:06:21.660207 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:21.660163    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:06:22.254614 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:22.254585    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:25.218183 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:25.218150    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:27.938373 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:27.938339    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:06:31.305465 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:31.305428    4955 generic.go:356] "Generic (PLEG): container finished" podID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerID="0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3" exitCode=0
Nov 03 16:06:31.305921 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:31.305503    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" event={"ID":"a3c63d7e-714f-4a32-8be9-9925d9a66050","Type":"ContainerDied","Data":"0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"}
Nov 03 16:06:31.305921 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:31.305895    4955 scope.go:122] "RemoveContainer" containerID="0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"
Nov 03 16:06:31.307234 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:31.307200    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:31.307294 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:31.307276    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:31.307330 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:31.307307    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:32.311808 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:32.311769    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" event={"ID":"a3c63d7e-714f-4a32-8be9-9925d9a66050","Type":"ContainerStarted","Data":"17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3"}
Nov 03 16:06:32.584466 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:32.584378    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:32.584643 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:32.584623    4955 patch_prober.go:29] interesting pod/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 03 16:06:32.584702 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:32.584684    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 03 16:06:33.585251 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:33.585215    4955 patch_prober.go:29] interesting pod/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 03 16:06:33.585766 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:33.585270    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 03 16:06:34.584548 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:34.584513    4955 patch_prober.go:29] interesting pod/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 03 16:06:34.584741 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:34.584583    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 03 16:06:35.584668 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:35.584635    4955 patch_prober.go:29] interesting pod/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 03 16:06:35.585139 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:35.584697    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 03 16:06:36.585366 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:36.585325    4955 patch_prober.go:29] interesting pod/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 03 16:06:36.585796 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:36.585380    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 03 16:06:37.588433 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:37.588401    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:37.588918 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:37.588851    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:37.589647 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:37.589625    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:06:46.424468 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:46.424427    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"]
Nov 03 16:06:51.425270 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.425204    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc" podUID="b307eb8c-a013-4b33-96c9-c9d925cc394e" containerName="payload-pre-processing" containerID="cri-o://06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05" gracePeriod=25
Nov 03 16:06:51.673306 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.673283    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:51.715537 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.715462    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume\") pod \"b307eb8c-a013-4b33-96c9-c9d925cc394e\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") "
Nov 03 16:06:51.715700 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.715588    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp\" (UniqueName: \"kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp\") pod \"b307eb8c-a013-4b33-96c9-c9d925cc394e\" (UID: \"b307eb8c-a013-4b33-96c9-c9d925cc394e\") "
Nov 03 16:06:51.715843 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.715820    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume" pod "b307eb8c-a013-4b33-96c9-c9d925cc394e" (UID: "b307eb8c-a013-4b33-96c9-c9d925cc394e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:06:51.715933 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.715919    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b307eb8c-a013-4b33-96c9-c9d925cc394e-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:51.717821 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.717800    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp" pod "b307eb8c-a013-4b33-96c9-c9d925cc394e" (UID: "b307eb8c-a013-4b33-96c9-c9d925cc394e"). InnerVolumeSpecName "kube-api-access-4g8gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:06:51.817207 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:51.817175    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8gp\" (UniqueName: \"kubernetes.io/projected/b307eb8c-a013-4b33-96c9-c9d925cc394e-kube-api-access-4g8gp\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:52.415198 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.415161    4955 generic.go:356] "Generic (PLEG): container finished" podID="b307eb8c-a013-4b33-96c9-c9d925cc394e" containerID="06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05" exitCode=0
Nov 03 16:06:52.415393 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.415212    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"
Nov 03 16:06:52.415393 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.415242    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc" event={"ID":"b307eb8c-a013-4b33-96c9-c9d925cc394e","Type":"ContainerDied","Data":"06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05"}
Nov 03 16:06:52.415393 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.415296    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc" event={"ID":"b307eb8c-a013-4b33-96c9-c9d925cc394e","Type":"ContainerDied","Data":"e03c959cc4b997427a240fe614dd46109e6031f3f1c263e2f45b0a54d0663a1d"}
Nov 03 16:06:52.415393 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.415319    4955 scope.go:122] "RemoveContainer" containerID="06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05"
Nov 03 16:06:52.424422 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.424403    4955 scope.go:122] "RemoveContainer" containerID="06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05"
Nov 03 16:06:52.424705 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:52.424683    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05\": container with ID starting with 06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05 not found: ID does not exist" containerID="06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05"
Nov 03 16:06:52.424749 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.424717    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05"} err="failed to get container status \"06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05\": rpc error: code = NotFound desc = could not find container \"06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05\": container with ID starting with 06d37bd20a7e0eaa571e0aface4fbf15c519999e29b32bfbaee2b49b52ecab05 not found: ID does not exist"
Nov 03 16:06:52.586812 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.586776    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"]
Nov 03 16:06:52.636435 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.636399    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cca28a2-d44cbfb87-4c8xc"]
Nov 03 16:06:52.850554 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.850462    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"]
Nov 03 16:06:52.850726 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:52.850699    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" podUID="6bd3de93-53b6-4281-9bd7-bb26a0a14067" containerName="maas-api" containerID="cri-o://5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2" gracePeriod=30
Nov 03 16:06:53.134994 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.133950    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:53.204786 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.204739    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b307eb8c-a013-4b33-96c9-c9d925cc394e" path="/var/lib/kubelet/pods/b307eb8c-a013-4b33-96c9-c9d925cc394e/volumes"
Nov 03 16:06:53.236195 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.236160    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp\" (UniqueName: \"kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp\") pod \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") "
Nov 03 16:06:53.236398 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.236227    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca\") pod \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") "
Nov 03 16:06:53.236398 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.236303    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs\") pod \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") "
Nov 03 16:06:53.236398 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.236365    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\") pod \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\" (UID: \"6bd3de93-53b6-4281-9bd7-bb26a0a14067\") "
Nov 03 16:06:53.236702 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.236674    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca" pod "6bd3de93-53b6-4281-9bd7-bb26a0a14067" (UID: "6bd3de93-53b6-4281-9bd7-bb26a0a14067"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:06:53.236917 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.236883    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd3de93-53b6-4281-9bd7-bb26a0a14067-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:53.238660 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.238637    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp" pod "6bd3de93-53b6-4281-9bd7-bb26a0a14067" (UID: "6bd3de93-53b6-4281-9bd7-bb26a0a14067"). InnerVolumeSpecName "kube-api-access-zbjwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:06:53.239175 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.239155    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs" pod "6bd3de93-53b6-4281-9bd7-bb26a0a14067" (UID: "6bd3de93-53b6-4281-9bd7-bb26a0a14067"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:06:53.239244 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.239173    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls" pod "6bd3de93-53b6-4281-9bd7-bb26a0a14067" (UID: "6bd3de93-53b6-4281-9bd7-bb26a0a14067"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:06:53.338153 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.338113    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjwp\" (UniqueName: \"kubernetes.io/projected/6bd3de93-53b6-4281-9bd7-bb26a0a14067-kube-api-access-zbjwp\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:53.338153 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.338144    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:53.338153 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.338155    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd3de93-53b6-4281-9bd7-bb26a0a14067-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:06:53.421416 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.421382    4955 generic.go:356] "Generic (PLEG): container finished" podID="6bd3de93-53b6-4281-9bd7-bb26a0a14067" containerID="5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2" exitCode=0
Nov 03 16:06:53.421632 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.421464    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" event={"ID":"6bd3de93-53b6-4281-9bd7-bb26a0a14067","Type":"ContainerDied","Data":"5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2"}
Nov 03 16:06:53.421632 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.421534    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt" event={"ID":"6bd3de93-53b6-4281-9bd7-bb26a0a14067","Type":"ContainerDied","Data":"cc2ae7a966a55109be8614544ce50f4094ee2cb966a0cb922dea74fa95e2d5a9"}
Nov 03 16:06:53.421632 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.421479    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"
Nov 03 16:06:53.421632 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.421563    4955 scope.go:122] "RemoveContainer" containerID="5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2"
Nov 03 16:06:53.433173 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.433152    4955 scope.go:122] "RemoveContainer" containerID="5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2"
Nov 03 16:06:53.433571 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:53.433545    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2\": container with ID starting with 5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2 not found: ID does not exist" containerID="5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2"
Nov 03 16:06:53.433675 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.433573    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2"} err="failed to get container status \"5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2\": rpc error: code = NotFound desc = could not find container \"5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2\": container with ID starting with 5ffd878c4e64bb201a1040450aebb53c71174e891f9aeb46a51be01794427af2 not found: ID does not exist"
Nov 03 16:06:53.591126 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.591097    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"]
Nov 03 16:06:53.612582 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:53.612550    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2cca28a2-cf7764755-rk8rt"]
Nov 03 16:06:55.203786 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:55.203752    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd3de93-53b6-4281-9bd7-bb26a0a14067" path="/var/lib/kubelet/pods/6bd3de93-53b6-4281-9bd7-bb26a0a14067/volumes"
Nov 03 16:06:56.105417 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:56.105375    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"]
Nov 03 16:06:56.105650 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:56.105621    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy" containerID="cri-o://17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3" gracePeriod=30
Nov 03 16:06:57.948732 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:06:57.948691    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:06:58.217738 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.217660    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"]
Nov 03 16:06:58.218081 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218068    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b307eb8c-a013-4b33-96c9-c9d925cc394e" containerName="payload-pre-processing"
Nov 03 16:06:58.218132 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218082    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b307eb8c-a013-4b33-96c9-c9d925cc394e" containerName="payload-pre-processing"
Nov 03 16:06:58.218132 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218100    4955 cpu_manager.go:414] "containerMap: removing container" podUID="6bd3de93-53b6-4281-9bd7-bb26a0a14067" containerName="maas-api"
Nov 03 16:06:58.218132 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218105    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd3de93-53b6-4281-9bd7-bb26a0a14067" containerName="maas-api"
Nov 03 16:06:58.218132 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218123    4955 cpu_manager.go:414] "containerMap: removing container" podUID="b728134d-dd38-443e-8c8f-6057caa22bac" containerName="maas-api"
Nov 03 16:06:58.218132 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218128    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b728134d-dd38-443e-8c8f-6057caa22bac" containerName="maas-api"
Nov 03 16:06:58.218297 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218185    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b307eb8c-a013-4b33-96c9-c9d925cc394e" containerName="payload-pre-processing"
Nov 03 16:06:58.218297 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218194    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd3de93-53b6-4281-9bd7-bb26a0a14067" containerName="maas-api"
Nov 03 16:06:58.218297 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.218204    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="b728134d-dd38-443e-8c8f-6057caa22bac" containerName="maas-api"
Nov 03 16:06:58.222629 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.222608    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.282228 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.282195    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-877924-openshift-default-dockercfg-zdclj\""
Nov 03 16:06:58.298386 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298350    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhj54\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298573 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298480    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298573 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298521    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298708 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298639    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298764 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298712    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298764 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298748    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298880 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298810    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298941 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298876    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.298941 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.298898    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.335214 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.334963    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"]
Nov 03 16:06:58.399430 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399397    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399430 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399433    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399690 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399454    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhj54\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399690 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399499    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399690 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399514    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399690 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399556    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399702    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399761    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399836    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399877    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.399947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.399897    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.400258 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.400164    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.400258 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.400184    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.400528 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.400284    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.402090 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.402067    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.402258 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.402230    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.462632 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.462606    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhj54\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.463497 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.463471    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token\") pod \"e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") " pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.535456 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.535381    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:58.980124 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.980096    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"]
Nov 03 16:06:58.982458 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:06:58.982429    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77c4a12_16b5_40b0_94f0_e0c57def7b7e.slice/crio-44de54bdb2cdc7501c2eb2e6cefa9b20e7ab3ff8a78f0e3a18cd820cdf9268bb WatchSource:0}: Error finding container 44de54bdb2cdc7501c2eb2e6cefa9b20e7ab3ff8a78f0e3a18cd820cdf9268bb: Status 404 returned error can't find the container with id 44de54bdb2cdc7501c2eb2e6cefa9b20e7ab3ff8a78f0e3a18cd820cdf9268bb
Nov 03 16:06:58.985025 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.984990    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:58.985139 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.985071    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:58.985139 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:58.985119    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:06:59.391532 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:59.391502    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:06:59.454727 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:59.454689    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" event={"ID":"e77c4a12-16b5-40b0-94f0-e0c57def7b7e","Type":"ContainerStarted","Data":"d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"}
Nov 03 16:06:59.454895 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:59.454741    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" event={"ID":"e77c4a12-16b5-40b0-94f0-e0c57def7b7e","Type":"ContainerStarted","Data":"44de54bdb2cdc7501c2eb2e6cefa9b20e7ab3ff8a78f0e3a18cd820cdf9268bb"}
Nov 03 16:06:59.536702 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:59.536671    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:06:59.589423 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:06:59.589371    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podStartSLOduration=1.589357506 podStartE2EDuration="1.589357506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:06:59.588367235 +0000 UTC m=+3534.513185149" watchObservedRunningTime="2026-11-03 16:06:59.589357506 +0000 UTC m=+3534.514175420"
Nov 03 16:07:00.541172 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:00.541145    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:01.358825 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.358799    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:07:01.434942 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.434913    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.434993    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435029    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435057    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435088    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435154 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435146    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435433 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435242    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435433 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435254    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:07:01.435433 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435284    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435433 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435337    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data\") pod \"a3c63d7e-714f-4a32-8be9-9925d9a66050\" (UID: \"a3c63d7e-714f-4a32-8be9-9925d9a66050\") "
Nov 03 16:07:01.435654 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435542    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:07:01.435654 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435607    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:07:01.435767 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435661    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:07:01.435828 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435779    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:07:01.435828 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435783    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.435949 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435826    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.435949 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435843    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.435949 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.435858    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c63d7e-714f-4a32-8be9-9925d9a66050-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.437552 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.437527    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "kube-api-access-v6lll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:07:01.438026 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.438007    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:07:01.438139 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.438095    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:07:01.438188 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.438140    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token" pod "a3c63d7e-714f-4a32-8be9-9925d9a66050" (UID: "a3c63d7e-714f-4a32-8be9-9925d9a66050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:07:01.466686 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.466655    4955 generic.go:356] "Generic (PLEG): container finished" podID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerID="17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3" exitCode=0
Nov 03 16:07:01.466849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.466706    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"
Nov 03 16:07:01.466849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.466736    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" event={"ID":"a3c63d7e-714f-4a32-8be9-9925d9a66050","Type":"ContainerDied","Data":"17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3"}
Nov 03 16:07:01.466849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.466788    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm" event={"ID":"a3c63d7e-714f-4a32-8be9-9925d9a66050","Type":"ContainerDied","Data":"60a4ec4ee6181a36922085fcc86660109c702f9c40c8345e28ef9dafb8ed9478"}
Nov 03 16:07:01.466849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.466810    4955 scope.go:122] "RemoveContainer" containerID="17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3"
Nov 03 16:07:01.467209 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.467181    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:01.468329 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.468307    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:01.477122 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.477100    4955 scope.go:122] "RemoveContainer" containerID="0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"
Nov 03 16:07:01.485847 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.485829    4955 scope.go:122] "RemoveContainer" containerID="17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3"
Nov 03 16:07:01.486142 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:07:01.486111    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3\": container with ID starting with 17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3 not found: ID does not exist" containerID="17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3"
Nov 03 16:07:01.486246 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.486151    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3"} err="failed to get container status \"17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3\": rpc error: code = NotFound desc = could not find container \"17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3\": container with ID starting with 17ce9b6ce4e47f7a5f418271715f46caa9f9ef9ace741035027a93c17ab9b6b3 not found: ID does not exist"
Nov 03 16:07:01.486246 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.486169    4955 scope.go:122] "RemoveContainer" containerID="0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"
Nov 03 16:07:01.486412 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:07:01.486395    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3\": container with ID starting with 0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3 not found: ID does not exist" containerID="0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"
Nov 03 16:07:01.486453 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.486424    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3"} err="failed to get container status \"0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3\": rpc error: code = NotFound desc = could not find container \"0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3\": container with ID starting with 0555c6f690050e04ca742894a8423303b570a86d7089fb70854741e3ef2571f3 not found: ID does not exist"
Nov 03 16:07:01.537152 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.537098    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.537152 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.537130    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.537152 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.537155    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.537397 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.537168    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6lll\" (UniqueName: \"kubernetes.io/projected/a3c63d7e-714f-4a32-8be9-9925d9a66050-kube-api-access-v6lll\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.537397 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.537179    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c63d7e-714f-4a32-8be9-9925d9a66050-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:07:01.799474 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.799395    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"]
Nov 03 16:07:01.803398 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:01.803374    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2cca28a2-openshift-default-7f74668549-scmzm"]
Nov 03 16:07:03.204845 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:03.204811    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" path="/var/lib/kubelet/pods/a3c63d7e-714f-4a32-8be9-9925d9a66050/volumes"
Nov 03 16:07:06.175634 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.175596    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"]
Nov 03 16:07:06.176058 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.176021    4955 cpu_manager.go:414] "containerMap: removing container" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy"
Nov 03 16:07:06.176058 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.176032    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy"
Nov 03 16:07:06.176137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.176063    4955 cpu_manager.go:414] "containerMap: removing container" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy"
Nov 03 16:07:06.176137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.176068    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy"
Nov 03 16:07:06.176137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.176131    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy"
Nov 03 16:07:06.180562 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.180546    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.239611 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.239571    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"]
Nov 03 16:07:06.287434 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.287395    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjmp\" (UniqueName: \"kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp\") pod \"payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.287924 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.287896    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.288193 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.288132    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-877924\""
Nov 03 16:07:06.288257 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.288245    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-877924-dockercfg-j8mqq\""
Nov 03 16:07:06.340570 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.340530    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"]
Nov 03 16:07:06.341185 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.341165    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c63d7e-714f-4a32-8be9-9925d9a66050" containerName="istio-proxy"
Nov 03 16:07:06.345568 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.345544    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.378712 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.378676    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"]
Nov 03 16:07:06.388763 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.388735    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjmp\" (UniqueName: \"kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp\") pod \"payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.388947 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.388819    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.389513 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.389488    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.451356 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.451288    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjmp\" (UniqueName: \"kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp\") pod \"payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.489475 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.489441    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wklqt\" (UniqueName: \"kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt\") pod \"payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.489647 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.489605    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.493620 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.493599    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:06.590640 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.590602    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.590831 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.590721    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wklqt\" (UniqueName: \"kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt\") pod \"payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.591318 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.591292    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.653211 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.653184    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wklqt\" (UniqueName: \"kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt\") pod \"payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.658580 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.658557    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:06.876432 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:06.876408    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"]
Nov 03 16:07:06.878481 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:07:06.878453    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20886cb7_0a22_4650_bc86_f525b8b420c8.slice/crio-fb7e3d134c09d35b5dac4577fbd6cd3aee143e70b4912afad80947375f78b120 WatchSource:0}: Error finding container fb7e3d134c09d35b5dac4577fbd6cd3aee143e70b4912afad80947375f78b120: Status 404 returned error can't find the container with id fb7e3d134c09d35b5dac4577fbd6cd3aee143e70b4912afad80947375f78b120
Nov 03 16:07:07.032275 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.032246    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"]
Nov 03 16:07:07.034425 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:07:07.034400    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f7d594_5ec8_4b0b_a8d9_e4a612321aa1.slice/crio-02768fd5f07cb1bea69fe5a403033a1ad17ceb1da791eeeb6357315bc0145f08 WatchSource:0}: Error finding container 02768fd5f07cb1bea69fe5a403033a1ad17ceb1da791eeeb6357315bc0145f08: Status 404 returned error can't find the container with id 02768fd5f07cb1bea69fe5a403033a1ad17ceb1da791eeeb6357315bc0145f08
Nov 03 16:07:07.499104 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.499064    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm" event={"ID":"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1","Type":"ContainerStarted","Data":"81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619"}
Nov 03 16:07:07.499560 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.499173    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:07.499560 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.499205    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm" event={"ID":"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1","Type":"ContainerStarted","Data":"02768fd5f07cb1bea69fe5a403033a1ad17ceb1da791eeeb6357315bc0145f08"}
Nov 03 16:07:07.500533 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.500495    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd" event={"ID":"20886cb7-0a22-4650-bc86-f525b8b420c8","Type":"ContainerStarted","Data":"1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516"}
Nov 03 16:07:07.500660 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.500548    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd" event={"ID":"20886cb7-0a22-4650-bc86-f525b8b420c8","Type":"ContainerStarted","Data":"fb7e3d134c09d35b5dac4577fbd6cd3aee143e70b4912afad80947375f78b120"}
Nov 03 16:07:07.500660 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.500575    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:07.618046 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.617989    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm" podStartSLOduration=1.617949751 podStartE2EDuration="1.617949751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:07:07.615940747 +0000 UTC m=+3542.540758660" watchObservedRunningTime="2026-11-03 16:07:07.617949751 +0000 UTC m=+3542.542767665"
Nov 03 16:07:07.735875 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:07.735775    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd" podStartSLOduration=1.7357577339999999 podStartE2EDuration="1.735757734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:07:07.733274693 +0000 UTC m=+3542.658092606" watchObservedRunningTime="2026-11-03 16:07:07.735757734 +0000 UTC m=+3542.660575729"
Nov 03 16:07:09.606282 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.606246    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"]
Nov 03 16:07:09.609846 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.609830    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.668840 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.668806    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-877924-serving-cert\""
Nov 03 16:07:09.721239 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.721207    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"]
Nov 03 16:07:09.721617 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.721596    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.721665 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.721640    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.721749 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.721735    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vccw\" (UniqueName: \"kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.721816 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.721803    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.823213 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.823172    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vccw\" (UniqueName: \"kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.823420 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.823276    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.823420 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.823325    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.823420 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.823366    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.823595 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:07:09.823495    4955 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-serving-cert: secret "maas-api-e2e-authcfg-http-877924-serving-cert" not found
Nov 03 16:07:09.823595 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:07:09.823585    4955 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls podName:f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b nodeName:}" failed. No retries permitted until 2026-11-03 16:07:10.323556636 +0000 UTC m=+3545.248374543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls") pod "maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" (UID: "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b") : secret "maas-api-e2e-authcfg-http-877924-serving-cert" not found
Nov 03 16:07:09.824044 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.824021    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.825943 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.825923    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:09.887907 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:09.887831    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vccw\" (UniqueName: \"kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:10.328161 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:10.328125    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:10.330847 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:10.330822    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:10.522750 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:10.522716    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:10.900021 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:10.899990    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"]
Nov 03 16:07:10.902072 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:07:10.902036    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d2f50c_1835_4be7_88b9_d9a2aa1b0c8b.slice/crio-dd1d0b59b983df4d32671b2a8fc614c001193c6d71782bc0dc5ed3f8a1398f5a WatchSource:0}: Error finding container dd1d0b59b983df4d32671b2a8fc614c001193c6d71782bc0dc5ed3f8a1398f5a: Status 404 returned error can't find the container with id dd1d0b59b983df4d32671b2a8fc614c001193c6d71782bc0dc5ed3f8a1398f5a
Nov 03 16:07:11.519507 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:11.519469    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" event={"ID":"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b","Type":"ContainerStarted","Data":"dd1d0b59b983df4d32671b2a8fc614c001193c6d71782bc0dc5ed3f8a1398f5a"}
Nov 03 16:07:12.529930 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:12.529872    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" event={"ID":"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b","Type":"ContainerStarted","Data":"5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0"}
Nov 03 16:07:12.530453 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:12.529955    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:12.658328 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:12.658276    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" podStartSLOduration=2.761194928 podStartE2EDuration="3.658261612s" totalImagesPullingTime="897.066684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:07:12.656584348 +0000 UTC m=+3547.581402264" watchObservedRunningTime="2026-11-03 16:07:12.658261612 +0000 UTC m=+3547.583079546"
Nov 03 16:07:17.970158 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:17.970120    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:07:18.507721 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:18.507693    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:07:18.507908 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:18.507765    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:07:18.539672 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:18.539644    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:07:27.607464 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:27.607430    4955 generic.go:356] "Generic (PLEG): container finished" podID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerID="d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537" exitCode=0
Nov 03 16:07:27.607901 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:27.607509    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" event={"ID":"e77c4a12-16b5-40b0-94f0-e0c57def7b7e","Type":"ContainerDied","Data":"d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"}
Nov 03 16:07:27.607950 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:27.607908    4955 scope.go:122] "RemoveContainer" containerID="d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"
Nov 03 16:07:27.609199 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:27.609167    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:07:27.609297 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:27.609246    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:07:27.609297 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:27.609277    4955 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 16:07:27.941500 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:07:27.941457    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:07:28.615569 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:28.615531    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" event={"ID":"e77c4a12-16b5-40b0-94f0-e0c57def7b7e","Type":"ContainerStarted","Data":"be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102"}
Nov 03 16:07:29.547995 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:29.543338    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:29.550648 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:29.550595    4955 patch_prober.go:29] interesting pod/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 03 16:07:29.550948 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:29.550896    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 03 16:07:30.554139 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:30.554098    4955 patch_prober.go:29] interesting pod/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 03 16:07:30.554827 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:30.554777    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 03 16:07:31.468206 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:31.468173    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:31.546064 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:31.545346    4955 patch_prober.go:29] interesting pod/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 03 16:07:31.546064 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:31.545424    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 03 16:07:32.541539 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:32.541501    4955 patch_prober.go:29] interesting pod/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 03 16:07:32.542196 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:32.542157    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 03 16:07:33.537863 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:33.537103    4955 patch_prober.go:29] interesting pod/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 03 16:07:33.537863 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:33.537163    4955 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 03 16:07:34.540285 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:34.540255    4955 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:34.541324 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:07:34.541306    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:07:57.938259 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:07:57.938209    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:08:05.948251 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:08:05.948214    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 16:08:05.956431 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:08:05.956411    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 16:08:05.958370 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:08:05.958351    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 16:08:05.966335 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:08:05.966316    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 16:08:27.938852 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:08:27.938814    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:08:57.937641 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:08:57.937603    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:09:00.006136 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:00.006101    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"]
Nov 03 16:09:00.114527 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:00.114490    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"]
Nov 03 16:09:05.006770 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.006719    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm" podUID="41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" containerName="payload-processing" containerID="cri-o://81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619" gracePeriod=25
Nov 03 16:09:05.115166 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.115116    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd" podUID="20886cb7-0a22-4650-bc86-f525b8b420c8" containerName="payload-pre-processing" containerID="cri-o://1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516" gracePeriod=25
Nov 03 16:09:05.272288 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.272224    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:09:05.326434 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.326403    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume\") pod \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") "
Nov 03 16:09:05.326631 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.326458    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt\" (UniqueName: \"kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt\") pod \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\" (UID: \"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1\") "
Nov 03 16:09:05.326882 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.326796    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume" pod "41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" (UID: "41f7d594-5ec8-4b0b-a8d9-e4a612321aa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:09:05.328874 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.328843    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt" pod "41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" (UID: "41f7d594-5ec8-4b0b-a8d9-e4a612321aa1"). InnerVolumeSpecName "kube-api-access-wklqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:09:05.389782 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.389695    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:09:05.427793 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.427762    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume\") pod \"20886cb7-0a22-4650-bc86-f525b8b420c8\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") "
Nov 03 16:09:05.427985 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.427851    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp\" (UniqueName: \"kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp\") pod \"20886cb7-0a22-4650-bc86-f525b8b420c8\" (UID: \"20886cb7-0a22-4650-bc86-f525b8b420c8\") "
Nov 03 16:09:05.428118 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.428104    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:05.428160 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.428121    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wklqt\" (UniqueName: \"kubernetes.io/projected/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1-kube-api-access-wklqt\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:05.428293 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.428260    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume" pod "20886cb7-0a22-4650-bc86-f525b8b420c8" (UID: "20886cb7-0a22-4650-bc86-f525b8b420c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:09:05.430175 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.430151    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp" pod "20886cb7-0a22-4650-bc86-f525b8b420c8" (UID: "20886cb7-0a22-4650-bc86-f525b8b420c8"). InnerVolumeSpecName "kube-api-access-mrjmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:09:05.529474 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.529384    4955 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20886cb7-0a22-4650-bc86-f525b8b420c8-plugins-config-volume\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:05.529474 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:05.529414    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrjmp\" (UniqueName: \"kubernetes.io/projected/20886cb7-0a22-4650-bc86-f525b8b420c8-kube-api-access-mrjmp\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:06.111703 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.111667    4955 generic.go:356] "Generic (PLEG): container finished" podID="41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" containerID="81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619" exitCode=0
Nov 03 16:09:06.112179 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.111716    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"
Nov 03 16:09:06.112179 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.111752    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm" event={"ID":"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1","Type":"ContainerDied","Data":"81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619"}
Nov 03 16:09:06.112179 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.111804    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm" event={"ID":"41f7d594-5ec8-4b0b-a8d9-e4a612321aa1","Type":"ContainerDied","Data":"02768fd5f07cb1bea69fe5a403033a1ad17ceb1da791eeeb6357315bc0145f08"}
Nov 03 16:09:06.112179 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.111831    4955 scope.go:122] "RemoveContainer" containerID="81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619"
Nov 03 16:09:06.113212 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.113120    4955 generic.go:356] "Generic (PLEG): container finished" podID="20886cb7-0a22-4650-bc86-f525b8b420c8" containerID="1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516" exitCode=0
Nov 03 16:09:06.113212 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.113169    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd" event={"ID":"20886cb7-0a22-4650-bc86-f525b8b420c8","Type":"ContainerDied","Data":"1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516"}
Nov 03 16:09:06.113212 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.113207    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd" event={"ID":"20886cb7-0a22-4650-bc86-f525b8b420c8","Type":"ContainerDied","Data":"fb7e3d134c09d35b5dac4577fbd6cd3aee143e70b4912afad80947375f78b120"}
Nov 03 16:09:06.113357 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.113175    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"
Nov 03 16:09:06.125521 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.125492    4955 scope.go:122] "RemoveContainer" containerID="81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619"
Nov 03 16:09:06.125960 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:06.125936    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619\": container with ID starting with 81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619 not found: ID does not exist" containerID="81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619"
Nov 03 16:09:06.126070 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.125994    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619"} err="failed to get container status \"81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619\": rpc error: code = NotFound desc = could not find container \"81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619\": container with ID starting with 81a71191bdb02d7ed5863b7b69a3d1ecffea773791f125195ab28b52b2e64619 not found: ID does not exist"
Nov 03 16:09:06.126070 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.126013    4955 scope.go:122] "RemoveContainer" containerID="1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516"
Nov 03 16:09:06.134208 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.134188    4955 scope.go:122] "RemoveContainer" containerID="1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516"
Nov 03 16:09:06.134474 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:06.134453    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516\": container with ID starting with 1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516 not found: ID does not exist" containerID="1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516"
Nov 03 16:09:06.134543 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.134485    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516"} err="failed to get container status \"1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516\": rpc error: code = NotFound desc = could not find container \"1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516\": container with ID starting with 1d334a71cba8e607493e6410c07931a32f6235b9f5b1d9b9cdb0e86a4b051516 not found: ID does not exist"
Nov 03 16:09:06.278148 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.278059    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"]
Nov 03 16:09:06.282764 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.282733    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-877924-5dc6f76959-qk2pd"]
Nov 03 16:09:06.393591 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.393552    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"]
Nov 03 16:09:06.397927 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:06.397899    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-877924-56fcdf74fc-nnzdm"]
Nov 03 16:09:07.204118 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:07.204089    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20886cb7-0a22-4650-bc86-f525b8b420c8" path="/var/lib/kubelet/pods/20886cb7-0a22-4650-bc86-f525b8b420c8/volumes"
Nov 03 16:09:07.204518 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:07.204503    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" path="/var/lib/kubelet/pods/41f7d594-5ec8-4b0b-a8d9-e4a612321aa1/volumes"
Nov 03 16:09:13.429143 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.429104    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"]
Nov 03 16:09:13.429643 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.429313    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" podUID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" containerName="maas-api" containerID="cri-o://5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0" gracePeriod=30
Nov 03 16:09:13.535107 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.535063    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" podUID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.153:8443/health\": dial tcp 10.132.0.153:8443: connect: connection refused"
Nov 03 16:09:13.708289 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.708260    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:09:13.816767 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.816734    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca" pod "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" (UID: "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:09:13.817073 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.816995    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca\") pod \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") "
Nov 03 16:09:13.819080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.819049    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs\") pod \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") "
Nov 03 16:09:13.819281 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.819120    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw\" (UniqueName: \"kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw\") pod \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") "
Nov 03 16:09:13.819281 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.819233    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\") pod \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\" (UID: \"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b\") "
Nov 03 16:09:13.820016 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.819632    4955 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-openshift-service-ca\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:13.838789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.835949    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls" pod "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" (UID: "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:09:13.840636 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.840575    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw" pod "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" (UID: "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b"). InnerVolumeSpecName "kube-api-access-4vccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:09:13.842332 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.842304    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs" pod "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" (UID: "f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:09:13.920626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.920589    4955 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-maas-api-tls\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:13.920626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.920625    4955 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-metrics-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:13.920840 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:13.920640    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vccw\" (UniqueName: \"kubernetes.io/projected/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b-kube-api-access-4vccw\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:14.152065 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.152031    4955 generic.go:356] "Generic (PLEG): container finished" podID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" containerID="5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0" exitCode=0
Nov 03 16:09:14.152259 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.152082    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" event={"ID":"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b","Type":"ContainerDied","Data":"5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0"}
Nov 03 16:09:14.152259 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.152089    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"
Nov 03 16:09:14.152259 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.152153    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn" event={"ID":"f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b","Type":"ContainerDied","Data":"dd1d0b59b983df4d32671b2a8fc614c001193c6d71782bc0dc5ed3f8a1398f5a"}
Nov 03 16:09:14.152259 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.152175    4955 scope.go:122] "RemoveContainer" containerID="5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0"
Nov 03 16:09:14.161902 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.161878    4955 scope.go:122] "RemoveContainer" containerID="5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0"
Nov 03 16:09:14.162304 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:14.162282    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0\": container with ID starting with 5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0 not found: ID does not exist" containerID="5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0"
Nov 03 16:09:14.162374 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.162316    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0"} err="failed to get container status \"5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0\": rpc error: code = NotFound desc = could not find container \"5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0\": container with ID starting with 5180ef181fe90da1d3946da87c63dc107c651b0ef504324747c38d496ebd52b0 not found: ID does not exist"
Nov 03 16:09:14.318915 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.318876    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"]
Nov 03 16:09:14.325742 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:14.325711    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-877924-67cddc8c9-8zxkn"]
Nov 03 16:09:15.206294 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:15.206248    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" path="/var/lib/kubelet/pods/f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b/volumes"
Nov 03 16:09:18.715514 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:18.715479    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:09:22.893256 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:22.893222    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"]
Nov 03 16:09:22.893766 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:22.893456    4955 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy" containerID="cri-o://be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102" gracePeriod=30
Nov 03 16:09:27.940165 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:27.940122    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:09:28.137898 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.137866    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:09:28.164404 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164331    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164404 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164370    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164404 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164393    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164635 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164427    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164635 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164476    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164635 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164521    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164635 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164594    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164651    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164677    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:09:28.164849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164724    4955 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy\") pod \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\" (UID: \"e77c4a12-16b5-40b0-94f0-e0c57def7b7e\") "
Nov 03 16:09:28.164849 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164751    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164874    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.164948    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.165056    4955 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.165070    4955 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-data\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.165087    4955 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istiod-ca-cert\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.165108    4955 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-credential-socket\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.165137 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.165114    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:09:28.166874 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.166843    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:09:28.167018 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.166894    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:09:28.167382 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.167362    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "kube-api-access-fhj54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:09:28.167382 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.167367    4955 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy" pod "e77c4a12-16b5-40b0-94f0-e0c57def7b7e" (UID: "e77c4a12-16b5-40b0-94f0-e0c57def7b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:09:28.225330 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.225297    4955 generic.go:356] "Generic (PLEG): container finished" podID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerID="be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102" exitCode=0
Nov 03 16:09:28.225533 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.225351    4955 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"
Nov 03 16:09:28.225533 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.225387    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" event={"ID":"e77c4a12-16b5-40b0-94f0-e0c57def7b7e","Type":"ContainerDied","Data":"be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102"}
Nov 03 16:09:28.225533 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.225449    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg" event={"ID":"e77c4a12-16b5-40b0-94f0-e0c57def7b7e","Type":"ContainerDied","Data":"44de54bdb2cdc7501c2eb2e6cefa9b20e7ab3ff8a78f0e3a18cd820cdf9268bb"}
Nov 03 16:09:28.225533 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.225480    4955 scope.go:122] "RemoveContainer" containerID="be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102"
Nov 03 16:09:28.234618 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.234599    4955 scope.go:122] "RemoveContainer" containerID="d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"
Nov 03 16:09:28.242788 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.242767    4955 scope.go:122] "RemoveContainer" containerID="be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102"
Nov 03 16:09:28.243062 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:28.243040    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102\": container with ID starting with be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102 not found: ID does not exist" containerID="be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102"
Nov 03 16:09:28.243148 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.243077    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102"} err="failed to get container status \"be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102\": rpc error: code = NotFound desc = could not find container \"be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102\": container with ID starting with be43a70c29a5433f532af467c08474210d5331b054bdba963771355ad6a85102 not found: ID does not exist"
Nov 03 16:09:28.243148 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.243106    4955 scope.go:122] "RemoveContainer" containerID="d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"
Nov 03 16:09:28.243372 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:28.243352    4955 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537\": container with ID starting with d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537 not found: ID does not exist" containerID="d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"
Nov 03 16:09:28.243414 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.243378    4955 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537"} err="failed to get container status \"d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537\": rpc error: code = NotFound desc = could not find container \"d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537\": container with ID starting with d10ad83f6604edaf8e2845843805614619ebe2e8499df24c2ebeaa2ebc465537 not found: ID does not exist"
Nov 03 16:09:28.266327 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.266298    4955 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhj54\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-kube-api-access-fhj54\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.266327 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.266324    4955 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-envoy\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.266327 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.266333    4955 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-podinfo\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.266518 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.266342    4955 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-workload-certs\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.266518 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.266350    4955 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77c4a12-16b5-40b0-94f0-e0c57def7b7e-istio-token\") on node \"ip-10-0-134-23.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 16:09:28.427957 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.427874    4955 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"]
Nov 03 16:09:28.432909 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:28.432881    4955 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-877924-openshift-default-8b54c6477-kddcg"]
Nov 03 16:09:29.204843 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:29.204805    4955 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" path="/var/lib/kubelet/pods/e77c4a12-16b5-40b0-94f0-e0c57def7b7e/volumes"
Nov 03 16:09:30.791866 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:30.791831    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:09:42.187714 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:09:42.187679    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:09:57.937637 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:09:57.937599    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:10:03.377113 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:03.377078    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:10:10.614994 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:10.614881    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-krrc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:10:10.614994 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:10.614945    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-krrc2" podUID="9c5727af-ba07-43b8-a640-da62782a09c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:10:10.726204 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:10.726154    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7w5hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:10:10.726430 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:10.726241    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7w5hv" podUID="932ea7e5-0b88-4fbf-a11a-1e8a34e6b677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:10:10.836292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:10.836132    4955 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-tq5bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:10:10.836292 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:10.836215    4955 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-tq5bn" podUID="92fdb587-5d2d-4d2e-af75-d69fade7bd20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:10:27.938820 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:10:27.938775    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:10:36.480457 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:36.480421    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-297t5"]
Nov 03 16:10:49.858433 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:49.858400    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7cx8f_d8c7575d-bac2-40c2-82ab-159007bfda0a/server/0.log"
Nov 03 16:10:51.813882 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:51.813850    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f5bf8854d-tgl9d_f2d298ca-33bd-4db0-a4fc-78955b570eff/maas-api/0.log"
Nov 03 16:10:56.359630 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:56.359596    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-297t5_131322e1-6527-4fe8-b6cb-c41284f51e0b/limitador/0.log"
Nov 03 16:10:57.938654 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:10:57.938608    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:10:58.026324 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:58.026288    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zmjhb_b240c106-868e-4be5-948f-a5072e182e28/discovery/0.log"
Nov 03 16:10:58.694383 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:58.694356    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fb7df4dc-bcxtb_508fd6a9-c488-4acc-a9f2-0ad7cd469837/kube-auth-proxy/0.log"
Nov 03 16:10:59.566886 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:59.566852    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-5zkfh_932b46a7-66e5-4991-ae83-a6086edb8cfe/istio-proxy/0.log"
Nov 03 16:10:59.914134 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:10:59.914105    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7w5hv_932ea7e5-0b88-4fbf-a11a-1e8a34e6b677/istio-proxy/0.log"
Nov 03 16:11:00.571650 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:00.571604    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-krrc2_9c5727af-ba07-43b8-a640-da62782a09c5/istio-proxy/0.log"
Nov 03 16:11:01.523019 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:01.522987    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-tq5bn_92fdb587-5d2d-4d2e-af75-d69fade7bd20/istio-proxy/0.log"
Nov 03 16:11:02.901660 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:02.901634    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b6f864dcc-6c9p5_9d77d0d5-bd4b-423f-b30a-994ad3a5e058/router/0.log"
Nov 03 16:11:04.181423 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:04.181385    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6_cff4535e-af16-4971-8c47-6777f461d121/main/0.log"
Nov 03 16:11:04.751144 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:04.751112    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lhbx9_44c5a3f3-e5c3-4f67-99de-5eb55e05bd06/main/0.log"
Nov 03 16:11:05.036686 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:05.036609    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-g4xf6_a4848f69-a0a9-4007-9203-7c656b86ef62/main/0.log"
Nov 03 16:11:05.845184 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:05.845154    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg_45c5a8e6-be22-4a5f-862e-c7d47d36f39a/main/0.log"
Nov 03 16:11:06.139304 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:06.139230    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt_958b5803-9348-4904-ac21-82254f5f9b90/main/0.log"
Nov 03 16:11:18.992576 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:18.992549    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7cx8f_d8c7575d-bac2-40c2-82ab-159007bfda0a/server/0.log"
Nov 03 16:11:20.608833 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:20.608799    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f5bf8854d-tgl9d_f2d298ca-33bd-4db0-a4fc-78955b570eff/maas-api/0.log"
Nov 03 16:11:24.499516 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:24.499477    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-297t5_131322e1-6527-4fe8-b6cb-c41284f51e0b/limitador/0.log"
Nov 03 16:11:25.780704 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:25.780666    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zmjhb_b240c106-868e-4be5-948f-a5072e182e28/discovery/0.log"
Nov 03 16:11:26.383212 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:26.383154    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fb7df4dc-bcxtb_508fd6a9-c488-4acc-a9f2-0ad7cd469837/kube-auth-proxy/0.log"
Nov 03 16:11:27.245792 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:27.245758    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-5zkfh_932b46a7-66e5-4991-ae83-a6086edb8cfe/istio-proxy/0.log"
Nov 03 16:11:27.533034 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:27.532933    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7w5hv_932ea7e5-0b88-4fbf-a11a-1e8a34e6b677/istio-proxy/0.log"
Nov 03 16:11:27.938569 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:11:27.938523    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:11:28.116250 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:28.116219    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-krrc2_9c5727af-ba07-43b8-a640-da62782a09c5/istio-proxy/0.log"
Nov 03 16:11:29.024071 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:29.024035    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-tq5bn_92fdb587-5d2d-4d2e-af75-d69fade7bd20/istio-proxy/0.log"
Nov 03 16:11:30.355172 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:30.355143    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b6f864dcc-6c9p5_9d77d0d5-bd4b-423f-b30a-994ad3a5e058/router/0.log"
Nov 03 16:11:31.037248 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:31.037215    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9mlm6_cff4535e-af16-4971-8c47-6777f461d121/main/0.log"
Nov 03 16:11:31.623032 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:31.622999    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lhbx9_44c5a3f3-e5c3-4f67-99de-5eb55e05bd06/main/0.log"
Nov 03 16:11:31.912984 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:31.912940    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-g4xf6_a4848f69-a0a9-4007-9203-7c656b86ef62/main/0.log"
Nov 03 16:11:32.741123 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:32.741086    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vcrwg_45c5a8e6-be22-4a5f-862e-c7d47d36f39a/main/0.log"
Nov 03 16:11:33.058288 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:11:33.058197    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-zjqwt_958b5803-9348-4904-ac21-82254f5f9b90/main/0.log"
Nov 03 16:11:57.937772 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:11:57.937731    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:12:27.938148 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:12:27.938107    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:12:57.937570 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:12:57.937529    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:13:05.853822 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:05.853790    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xnsmp_1e3decdb-1a61-4186-9f1d-c41cac5bb6da/global-pull-secret-syncer/0.log"
Nov 03 16:13:05.985201 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:05.985161    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 16:13:05.991355 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:05.991332    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-x66cq_a4e32724-e514-41cc-b175-59da7bfd7b8b/console-operator/2.log"
Nov 03 16:13:05.993789 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:05.993763    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 16:13:06.000080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:06.000060    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kzzmg_d116bb71-10c3-40c9-ac2d-50a79bc72946/ovn-acl-logging/0.log"
Nov 03 16:13:06.449075 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:06.449040    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gh68l_852144f2-7df8-4279-bf86-65c5351c8945/connection-checker/0.log"
Nov 03 16:13:06.874647 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:06.874572    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lxnsf_5e1df8e2-38c0-4e61-a117-f3b797ec963b/connection-checker/0.log"
Nov 03 16:13:07.313716 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:07.313677    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qb946_f50eed36-43f3-4f6d-8aa0-46a148ca8177/connection-checker/0.log"
Nov 03 16:13:08.218646 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:08.218609    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pq4lr_2afcc4bb-db5b-431a-bdb9-580af2eccd18/konnectivity-agent/0.log"
Nov 03 16:13:09.054486 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:09.054450    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-23.us-west-1.compute.internal_6cc7811c328f411b1828c6a10a8039cb/haproxy/0.log"
Nov 03 16:13:14.634245 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634157    4955 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"]
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634533    4955 cpu_manager.go:414] "containerMap: removing container" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634544    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634560    4955 cpu_manager.go:414] "containerMap: removing container" podUID="20886cb7-0a22-4650-bc86-f525b8b420c8" containerName="payload-pre-processing"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634565    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20886cb7-0a22-4650-bc86-f525b8b420c8" containerName="payload-pre-processing"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634574    4955 cpu_manager.go:414] "containerMap: removing container" podUID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" containerName="maas-api"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634579    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" containerName="maas-api"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634586    4955 cpu_manager.go:414] "containerMap: removing container" podUID="41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" containerName="payload-processing"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634591    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" containerName="payload-processing"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634611    4955 cpu_manager.go:414] "containerMap: removing container" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy"
Nov 03 16:13:14.634626 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634615    4955 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy"
Nov 03 16:13:14.635018 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634681    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77c4a12-16b5-40b0-94f0-e0c57def7b7e" containerName="istio-proxy"
Nov 03 16:13:14.635018 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634693    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="20886cb7-0a22-4650-bc86-f525b8b420c8" containerName="payload-pre-processing"
Nov 03 16:13:14.635018 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634701    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d2f50c-1835-4be7-88b9-d9a2aa1b0c8b" containerName="maas-api"
Nov 03 16:13:14.635018 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.634707    4955 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f7d594-5ec8-4b0b-a8d9-e4a612321aa1" containerName="payload-processing"
Nov 03 16:13:14.637640 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.637619    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.691620 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.691596    4955 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-76mwr\"/\"default-dockercfg-msnrf\""
Nov 03 16:13:14.738050 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.738016    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-proc\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.738240 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.738136    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-podres\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.738240 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.738188    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-sys\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.738326 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.738281    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-lib-modules\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.738326 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.738303    4955 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvl8\" (UniqueName: \"kubernetes.io/projected/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-kube-api-access-nbvl8\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.741594 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.741574    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76mwr\"/\"openshift-service-ca.crt\""
Nov 03 16:13:14.741708 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.741675    4955 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76mwr\"/\"kube-root-ca.crt\""
Nov 03 16:13:14.743804 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.743780    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"]
Nov 03 16:13:14.839787 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.839748    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-lib-modules\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.839787 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.839792    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvl8\" (UniqueName: \"kubernetes.io/projected/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-kube-api-access-nbvl8\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.839854    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-proc\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.839931    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-podres\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.839955    4955 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-sys\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.840056    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-proc\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840080 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.840068    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-podres\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840286 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.840097    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-sys\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.840286 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.840103    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-lib-modules\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.902233 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.902197    4955 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvl8\" (UniqueName: \"kubernetes.io/projected/ac68c54b-d66f-4cc0-ae48-d8ff1db54828-kube-api-access-nbvl8\") pod \"perf-node-gather-daemonset-9jk8p\" (UID: \"ac68c54b-d66f-4cc0-ae48-d8ff1db54828\") " pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:14.950072 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:14.950039    4955 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:15.326483 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:15.326454    4955 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"]
Nov 03 16:13:15.328677 ip-10-0-134-23 kubenswrapper[4955]: W1103 16:13:15.328650    4955 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac68c54b_d66f_4cc0_ae48_d8ff1db54828.slice/crio-ed05e58cd5e46bdc66db2d0ceef7636d14070524cc9c06be99622f3e9a3cc200 WatchSource:0}: Error finding container ed05e58cd5e46bdc66db2d0ceef7636d14070524cc9c06be99622f3e9a3cc200: Status 404 returned error can't find the container with id ed05e58cd5e46bdc66db2d0ceef7636d14070524cc9c06be99622f3e9a3cc200
Nov 03 16:13:15.402066 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:15.402031    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p" event={"ID":"ac68c54b-d66f-4cc0-ae48-d8ff1db54828","Type":"ContainerStarted","Data":"ed05e58cd5e46bdc66db2d0ceef7636d14070524cc9c06be99622f3e9a3cc200"}
Nov 03 16:13:16.408428 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:16.408393    4955 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p" event={"ID":"ac68c54b-d66f-4cc0-ae48-d8ff1db54828","Type":"ContainerStarted","Data":"ead47c1874b0bd72b3b2e5a6b8964160babd969fa8fd740e7f4472d8d3b556a7"}
Nov 03 16:13:16.408847 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:16.408525    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:16.529609 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:16.529543    4955 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p" podStartSLOduration=2.529525723 podStartE2EDuration="2.529525723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:16.528912515 +0000 UTC m=+3911.453730428" watchObservedRunningTime="2026-11-03 16:13:16.529525723 +0000 UTC m=+3911.454343637"
Nov 03 16:13:17.365846 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:17.365812    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-297t5_131322e1-6527-4fe8-b6cb-c41284f51e0b/limitador/0.log"
Nov 03 16:13:22.423409 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:22.423373    4955 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-76mwr/perf-node-gather-daemonset-9jk8p"
Nov 03 16:13:23.494897 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:23.494866    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-jlb5j_8db8c20d-d878-475b-97d0-6cd01befd3ae/cluster-monitoring-operator/0.log"
Nov 03 16:13:27.413394 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:27.413360    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7m9rk_a519036c-be04-44c3-925f-d1bb53ccbba1/node-exporter/0.log"
Nov 03 16:13:27.851653 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:27.851559    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7m9rk_a519036c-be04-44c3-925f-d1bb53ccbba1/kube-rbac-proxy/0.log"
Nov 03 16:13:27.943357 ip-10-0-134-23 kubenswrapper[4955]: E1103 16:13:27.943313    4955 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fnprj" containerName="extractor"
Nov 03 16:13:28.291963 ip-10-0-134-23 kubenswrapper[4955]: I1103 16:13:28.291932    4955 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7m9rk_a519036c-be04-44c3-925f-d1bb53ccbba1/init-textfile/0.log"