Nov 03 17:44:29.755014 ip-10-0-141-108 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:44:30.191483 ip-10-0-141-108 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:44:30.191483 ip-10-0-141-108 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:44:30.191483 ip-10-0-141-108 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:44:30.191483 ip-10-0-141-108 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:44:30.197251 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197174    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.197251 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197250    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.197251 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197254    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197257    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197260    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197263    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197266    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197268    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197272    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197274    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197277    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197280    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197282    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197285    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197295    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197298    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197301    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197304    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197306    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197309    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197312    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.197341 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197314    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197317    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197320    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197322    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197325    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197327    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197330    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197333    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197336    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197338    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197341    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197344    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197346    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197349    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197352    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197355    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197358    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197360    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197363    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197365    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.197838 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197368    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197370    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197372    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197375    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197377    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197380    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197383    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197385    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197388    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197390    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197393    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197395    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197400    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197405    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197408    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197411    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197414    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197417    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197420    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.198305 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197423    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197426    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197429    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197431    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197434    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197437    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197439    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197442    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197444    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197447    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197449    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197452    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197454    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197457    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197459    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197462    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197464    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197467    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197470    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197472    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.198798 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197475    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197477    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197480    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197483    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197486    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197490    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197495    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197498    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197501    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197503    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197505    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197508    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197510    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197514    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197516    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197519    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197522    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197525    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197527    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.199276 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197530    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197533    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197535    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197538    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197541    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197553    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.197557    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198369    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198377    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198380    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198383    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198386    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198389    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198392    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198400    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198403    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198406    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198409    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198411    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.199769 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198414    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198416    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198419    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198421    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198424    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198427    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198429    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198432    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198435    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198438    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198440    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198443    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198446    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198448    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198451    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198454    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198456    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198459    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198461    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198464    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.200231 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198466    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198469    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198471    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198474    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198477    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198479    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198482    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198484    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198492    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198495    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198498    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198500    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198503    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198505    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198509    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198511    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198514    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198516    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198519    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198521    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.200734 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198524    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198526    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198535    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198538    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198541    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198543    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198545    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198548    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198557    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198559    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198562    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198565    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198567    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198570    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198574    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198578    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198581    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198584    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198587    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.201215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198589    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198592    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198599    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198602    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198605    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198608    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198610    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198613    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198615    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198618    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198621    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198623    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198626    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198629    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198631    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198634    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198636    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198639    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198643    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.201710 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198646    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198649    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198652    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198655    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198657    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198660    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198662    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198665    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198668    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198671    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198673    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198676    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198678    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198681    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198683    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.198686    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199197    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199203    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199206    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.202186 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199208    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199214    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199217    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199219    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199222    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199225    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199230    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199232    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199235    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199238    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199240    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199243    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199245    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199248    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199251    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199254    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199256    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199259    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199261    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199264    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.202740 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199266    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199269    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199272    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199274    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199277    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199280    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199283    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199285    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199288    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199291    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199295    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199299    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199302    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199306    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199310    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199312    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199315    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199318    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199323    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.203215 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199325    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199328    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199330    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199333    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199335    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199338    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199340    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199343    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199346    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199348    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199351    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199353    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199356    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199358    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199361    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199364    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199366    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199369    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199371    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.203661 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199374    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199376    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199379    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199381    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199384    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199388    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199392    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199395    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199398    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199401    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199403    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199406    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199410    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199412    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199415    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199417    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199420    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199423    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199425    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.204128 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199428    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199430    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199433    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199435    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199437    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199440    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199443    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199445    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199448    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199450    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199453    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199456    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199458    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199461    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199463    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199466    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199468    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199471    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199473    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199476    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.204571 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199479    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199481    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199484    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199486    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199489    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.199493    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199590    2581 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199601    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199606    2581 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199611    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199615    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199618    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199623    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199627    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199630    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199633    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199637    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199640    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199643    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199646    2581 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199649    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199652    2581 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199655    2581 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199658    2581 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:44:30.205066 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199665    2581 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199668    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199671    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199674    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199678    2581 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199682    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199685    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199689    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199693    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199696    2581 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199699    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199702    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199705    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199709    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199713    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199728    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199731    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199734    2581 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199737    2581 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199740    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199744    2581 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199748    2581 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199751    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199754    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:44:30.205625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199757    2581 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199761    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199764    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199767    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199770    2581 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199773    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199776    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199779    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199782    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199787    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199790    2581 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199793    2581 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199797    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199800    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199803    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199807    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199810    2581 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199813    2581 flags.go:64] FLAG: --help="false"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199816    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199820    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199823    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199826    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199830    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:44:30.206259 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199833    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199836    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199839    2581 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199842    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199845    2581 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199848    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199851    2581 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199854    2581 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199857    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199860    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199863    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199865    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199868    2581 flags.go:64] FLAG: --lock-file=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199871    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199874    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199877    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199883    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199886    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199890    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199893    2581 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199896    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199899    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199902    2581 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199905    2581 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199909    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:44:30.207192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199912    2581 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199917    2581 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199920    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199923    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199926    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199929    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199932    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199935    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199938    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199945    2581 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199948    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199951    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199954    2581 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199957    2581 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199960    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199963    2581 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199966    2581 flags.go:64] FLAG: --port="10250"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199969    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199972    2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0eeb938a8f227e418"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199976    2581 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199979    2581 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199982    2581 flags.go:64] FLAG: --register-node="true"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199984    2581 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199988    2581 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199991    2581 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199995    2581 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:44:30.208205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.199998    2581 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200002    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200005    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200009    2581 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200012    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200015    2581 flags.go:64] FLAG: --runonce="false"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200018    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200021    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200024    2581 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200028    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200031    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200034    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200037    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200040    2581 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200043    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200045    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200049    2581 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200051    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200055    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200057    2581 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200060    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200065    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200068    2581 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200071    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200075    2581 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:44:30.208850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200078    2581 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200081    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200084    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200088    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200091    2581 flags.go:64] FLAG: --v="2"
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200095    2581 flags.go:64] FLAG: --version="false"
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200101    2581 flags.go:64] FLAG: --vmodule=""
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200104    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200108    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200241    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200245    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200248    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200251    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200254    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200257    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200260    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200263    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200266    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200269    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200272    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200274    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.209440 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200277    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200280    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200283    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200285    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200288    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200290    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200293    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200296    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200298    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200301    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200303    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200306    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200309    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200313    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200315    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200318    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200320    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200325    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200327    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200330    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.209968 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200333    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200336    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200338    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200341    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200343    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200346    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200348    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200351    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200354    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200357    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200359    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200362    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200364    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200367    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200369    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200372    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200374    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200377    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200379    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.210433 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200382    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200384    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200387    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200389    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200392    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200394    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200398    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200400    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200403    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200405    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200409    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200412    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200414    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200417    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200419    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200422    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200424    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200427    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200430    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.210908 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200432    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200435    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200437    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200440    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200443    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200447    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200451    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200455    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200458    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200461    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200464    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200466    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200469    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200472    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200475    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200477    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200479    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200482    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200484    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.211388 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200488    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200491    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200493    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200496    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200500    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200503    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200505    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200508    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200510    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200513    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200516    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200518    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200521    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200524    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200526    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200529    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.211934 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200531    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.200556    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200591    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200595    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200598    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200601    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200603    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200606    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200609    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200611    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200614    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200616    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200619    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200622    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200624    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200627    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.212298 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200631    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200634    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200636    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200639    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200642    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200647    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200650    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200652    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200655    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200657    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200660    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200663    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200666    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200668    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200671    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200673    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200676    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200678    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200681    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.212672 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200684    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200687    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200690    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200692    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200695    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200697    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200700    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200702    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200705    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200707    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200710    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200712    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200726    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200730    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200733    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200736    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200738    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200741    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200745    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.213143 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200748    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200751    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200753    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200756    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200759    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200762    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200767    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200771    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200774    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200777    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200780    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200783    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200785    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200789    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200792    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200794    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200797    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200799    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200802    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.213609 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200805    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200807    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200810    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200812    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200815    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200817    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200820    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200824    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200826    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200829    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200831    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200834    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200838    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200841    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200844    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200846    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200849    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200852    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200854    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200857    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.214098 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200859    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200862    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200864    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200867    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200870    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200872    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200875    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200878    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200880    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200883    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200887    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200889    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200892    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200894    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.200897    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.209448    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.209466    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.210991    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.210998    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:44:30.214561 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211063    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211069    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211072    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211077    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211081    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211085    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211089    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211092    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211095    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211099    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211102    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211104    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211107    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211110    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211112    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211115    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211118    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211121    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211123    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.215027 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211126    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211128    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211131    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211134    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211136    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211139    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211141    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211144    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211152    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211155    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211158    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211160    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211163    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211166    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211169    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211171    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211174    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211177    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211180    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.215464 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211182    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211185    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211187    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211190    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211192    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211195    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211198    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211200    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211203    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211206    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211209    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211211    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211214    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211216    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211219    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211221    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211224    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211227    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211229    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211232    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.216014 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211235    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211238    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211240    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211243    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211246    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211249    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211251    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211254    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211256    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211259    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211262    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211264    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211267    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211269    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211273    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211277    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211280    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211283    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211287    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.216481 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211289    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211292    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211295    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211298    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211301    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211303    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211306    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211308    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211311    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211313    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211316    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211319    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211322    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211324    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211327    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211329    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211333    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211336    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211338    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.216982 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211341    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211343    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211346    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211348    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211351    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211354    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211356    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211359    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211361    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211364    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.211391    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211552    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211557    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211560    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211563    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211566    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.217429 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211569    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211572    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211575    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211577    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211580    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211583    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211586    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211589    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211591    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211594    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211597    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211599    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211602    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211605    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211608    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211611    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211613    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211616    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211619    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.217840 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211621    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211624    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211627    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211629    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211633    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211636    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211639    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211642    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211644    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211647    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211650    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211652    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211655    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211658    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211661    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211665    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211669    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211672    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211675    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211678    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.218302 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211681    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211684    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211687    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211690    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211692    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211695    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211697    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211700    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211702    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211705    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211707    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211710    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211712    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211715    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211730    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211733    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211736    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211738    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211741    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211743    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.218844 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211746    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211748    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211751    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211753    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211756    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211758    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211761    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211764    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211767    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211769    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211772    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211774    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211777    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211779    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211782    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211785    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211788    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211791    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211793    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211795    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.219311 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211798    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211800    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211803    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211805    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211808    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211811    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211813    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211816    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211818    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211820    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211823    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211825    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211828    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211830    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211833    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211835    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211838    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211840    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211843    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211845    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.219784 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211849    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211852    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.211875    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211907    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211911    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211915    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211917    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211920    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211923    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211926    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211928    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211931    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211933    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211936    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:44:30.220250 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211940    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211944    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211947    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211950    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211953    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211956    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211959    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211962    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211964    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211967    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211970    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211974    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211977    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211980    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211983    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211985    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211987    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211990    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211993    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211996    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:44:30.220572 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.211999    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212001    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212004    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212007    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212009    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212012    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212014    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212016    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212019    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212022    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212025    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212027    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212030    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212032    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212034    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212037    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212039    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212042    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212044    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212047    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:44:30.221048 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212049    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212052    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212054    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212057    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212059    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212064    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212067    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212070    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212072    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212075    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212078    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212080    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212083    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212086    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212089    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212091    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212094    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212096    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212099    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212101    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:44:30.221500 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212104    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212107    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212109    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212112    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212114    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212117    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212119    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212122    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212124    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212127    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212129    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212132    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212134    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212137    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212139    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212142    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212145    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212148    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212151    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:44:30.222021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212153    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212156    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212158    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212161    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212164    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212166    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212170    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212172    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212175    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212177    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212180    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212183    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212186    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212189    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212191    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:30.212194    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.212251    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.214475    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.215483    2581 server.go:1014] "Starting client certificate rotation"
Nov 03 17:44:30.222468 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.215579    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:44:30.222932 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.215613    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:44:30.222932 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.217784    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:44:30.222932 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.218854    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:44:30.233913 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.233898    2581 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:44:30.237317 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.237301    2581 log.go:25] "Validated CRI v1 image API"
Nov 03 17:44:30.238163 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.238139    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:44:30.240600 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.240581    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9cd359b3-3316-45ae-af1a-4cbceff0a6ed:/dev/nvme0n1p4 fcf34f94-4228-4d45-8c3a-47b77eea6e2f:/dev/nvme0n1p3]
Nov 03 17:44:30.240659 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.240600    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 17:44:30.247840 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.247696    2581 manager.go:217] Machine: {Timestamp:2026-11-03 17:44:30.246165011 +0000 UTC m=+0.375161581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091526 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2267048449c1d682eed69ee08bc1b5 SystemUUID:ec226704-8449-c1d6-82ee-d69ee08bc1b5 BootID:326d59a0-c003-4215-8a45-21d902b1395a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d4:f0:36:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d4:f0:36:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:ba:5f:67:4a:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:44:30.247840 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.247829    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:44:30.248028 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.247933    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:44:30.249309 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.249281    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:44:30.249432 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.249312    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-108.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:44:30.249481 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.249435    2581 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:44:30.249481 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.249442    2581 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:44:30.249481 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.249472    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:44:30.249564 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.249487    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:44:30.250655 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.250644    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:44:30.254745 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.254728    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:44:30.254867 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.254858    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:44:30.255668 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.255658    2581 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:44:30.255703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.255671    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:44:30.255703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.255681    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:44:30.255703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.255690    2581 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:44:30.255703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.255698    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:44:30.256055 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.256044    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:44:30.258510 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.258494    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:44:30.264999 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.264982    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:44:30.265927 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.265911    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:44:30.266011 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.265954    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:44:30.266159 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266145    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:44:30.266217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266165    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:44:30.266217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266174    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:44:30.266217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266183    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:44:30.266217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266193    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:44:30.266217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266203    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:44:30.266922 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266910    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:44:30.266974 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266925    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:44:30.266974 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266938    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:44:30.266974 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266951    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:44:30.266974 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266965    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:44:30.267146 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.266980    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:44:30.268353 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.268341    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:44:30.268409 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.268357    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:44:30.272196 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.272179    2581 server.go:1263] "Started kubelet"
Nov 03 17:44:30.272282 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.272248    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:44:30.272343 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.272276    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:44:30.272383 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.272360    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:44:30.272902 ip-10-0-141-108 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:44:30.274118 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.274100    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:44:30.276469 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.276451    2581 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:44:30.280011 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.279987    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 17:44:30.282123 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.282109    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:44:30.282196 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.282133    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:44:30.282247 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.282215    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:30.283027 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283007    2581 factory.go:55] Registering systemd factory
Nov 03 17:44:30.283027 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283028    2581 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:44:30.283192 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283181    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:44:30.283244 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283180    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:44:30.283244 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283197    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:44:30.283335 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283316    2581 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:44:30.283335 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283319    2581 factory.go:153] Registering CRI-O factory
Nov 03 17:44:30.283404 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283336    2581 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:44:30.283404 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283398    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:44:30.283474 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283325    2581 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:44:30.283474 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283416    2581 factory.go:103] Registering Raw factory
Nov 03 17:44:30.283474 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.283430    2581 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:44:30.284785 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.284772    2581 manager.go:319] Starting recovery of all containers
Nov 03 17:44:30.298753 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.298703    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:44:30.298868 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.298766    2581 manager.go:324] Recovery completed
Nov 03 17:44:30.304274 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.304259    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.306464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.306447    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.306540 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.306485    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.306540 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.306500    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.306997 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.306983    2581 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:44:30.306997 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.306995    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:44:30.307089 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.307016    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:44:30.307166 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.306995    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.307843 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.307826    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.307895 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.307858    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.307895 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.307872    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.383203 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.383176    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:30.410244 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.410210    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-108.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 17:44:30.441104 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.441078    2581 policy_none.go:50] "Start"
Nov 03 17:44:30.441207 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.441116    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:44:30.441207 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.441133    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:44:30.443148 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.443114    2581 policy_none.go:44] "Start"
Nov 03 17:44:30.469888 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.454615    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-108.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 17:44:30.469888 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.454675    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-108.us-west-2.compute.internal.18e499fe934a2ebd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-108.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-108.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:30.272155325 +0000 UTC m=+0.401151896,LastTimestamp:2026-11-03 17:44:30.272155325 +0000 UTC m=+0.401151896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-108.us-west-2.compute.internal,}"
Nov 03 17:44:30.469888 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.467513    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:44:30.469888 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.467542    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:44:30.469888 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.467559    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:44:30.469888 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.467589    2581 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 17:44:30.473965 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.473937    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 17:44:30.474053 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.473937    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-108.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 17:44:30.475280 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.475251    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:44:30.479237 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479223    2581 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:44:30.479325 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.479249    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:44:30.479325 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479257    2581 server.go:87] "Starting device plugin registration server"
Nov 03 17:44:30.479456 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479445    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:44:30.479508 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479456    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:44:30.479601 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479586    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:44:30.479667 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479656    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:44:30.479780 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.479667    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:44:30.480014 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.479998    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:44:30.480072 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.480044    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:30.504160 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.504138    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:44:30.504239 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.504198    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pw4kw"
Nov 03 17:44:30.568515 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.568487    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"]
Nov 03 17:44:30.568611 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.568546    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.569552 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.569530    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.569603 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.569567    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.569603 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.569578    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.571912 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.571900    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.572069 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.572135 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572096    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.572626 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572610    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.572697 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572616    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.572697 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572659    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.572697 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572670    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.572697 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572640    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.572841 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.572706    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.574847 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.574832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.574925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.574858    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.575674 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.575656    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.575778 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.575679    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.575778 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.575688    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.580520 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.580507    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:30.581121 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.581108    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:30.581203 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.581130    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:30.581203 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.581141    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:30.581203 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.581158    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.585643 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.585626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acca05a04d25dcd6bb0ab5ae22e6769a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"acca05a04d25dcd6bb0ab5ae22e6769a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.585690 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.585648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.585690 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.585663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.609419 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.609398    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.616303 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.616283    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.652947 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.652920    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:30.655116 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.655100    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:30.656076 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.656062    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.656124 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.656092    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-108.us-west-2.compute.internal\": node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:30.686396 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.686376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.686505 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.686401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acca05a04d25dcd6bb0ab5ae22e6769a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"acca05a04d25dcd6bb0ab5ae22e6769a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.686505 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.686419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.686505 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.686469    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.686505 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.686490    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.686630 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.686492    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acca05a04d25dcd6bb0ab5ae22e6769a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"acca05a04d25dcd6bb0ab5ae22e6769a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.769998 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.769922    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:30.870809 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.870777    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:30.913767 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.913714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.921223 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:30.921206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:30.970946 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:30.970908    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.071474 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.071405    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.171610 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.171578    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.215846 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.215822    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 17:44:31.216420 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.215966    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:44:31.216420 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.216039    2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad185ca6b177f4c7286538684ef3f4ce-d7e05c9bd6f41af3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-108.us-west-2.compute.internal.18e499fe955652c0\": read tcp 10.0.141.108:52110->77.113.21.42:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-108.us-west-2.compute.internal.18e499fe955652c0  default   4746 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-108.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-108.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-108.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:30 +0000 UTC,LastTimestamp:2026-11-03 17:44:30.569581937 +0000 UTC m=+0.698578507,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-108.us-west-2.compute.internal,}"
Nov 03 17:44:31.272689 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.272655    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.374362 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.374280    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.474525 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.474494    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.510708 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.510653    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:39:30 +0000 UTC" deadline="2028-05-10 04:54:40.408387022 +0000 UTC"
Nov 03 17:44:31.510708 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.510707    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13283h10m8.897685261s"
Nov 03 17:44:31.514023 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.514009    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:44:31.515052 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.515025    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:44:31.515147 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.515066    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:44:31.515147 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.515082    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:44:31.577851 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.577830    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.678355 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.678279    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.778783 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.778740    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.878882 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:31.878846    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:44:31.882333 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.882315    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:31.882428 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.882387    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:31.952612 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.952577    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:44:31.953659 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:31.953631    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:44:32.026952 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.026922    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:44:32.060118 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.060081    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacca05a04d25dcd6bb0ab5ae22e6769a.slice/crio-989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea WatchSource:0}: Error finding container 989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea: Status 404 returned error can't find the container with id 989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea
Nov 03 17:44:32.064545 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.064511    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2eb497638654ab51d85f8c48e93d1a5.slice/crio-0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939 WatchSource:0}: Error finding container 0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939: Status 404 returned error can't find the container with id 0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939
Nov 03 17:44:32.067731 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.067698    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:44:32.093555 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.093530    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:32.258907 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.258810    2581 apiserver.go:52] "Watching apiserver"
Nov 03 17:44:32.283909 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.283872    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 17:44:32.435336 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.435308    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:44:32.435691 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.435669    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7mnmh","kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-m8nvv","openshift-multus/multus-additional-cni-plugins-nmrf9","openshift-network-diagnostics/network-check-target-jrt48","openshift-network-operator/iptables-alerter-hn245","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal","openshift-multus/multus-qkcgs","openshift-ovn-kubernetes/ovnkube-node-2crz8","kube-system/konnectivity-agent-j2vd6","openshift-image-registry/node-ca-8z24k","openshift-multus/network-metrics-daemon-8b682","kube-system/global-pull-secret-syncer-zc2rm"]
Nov 03 17:44:32.439765 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.439718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.442056 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.442041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.444552 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.444535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.447211 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.447186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.449873 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.449522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:32.449978 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.449959    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:32.453279 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.453258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.455646 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.455627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.458206 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.458186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.461029 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.461013    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.463371 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.463354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.465896 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.465877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:32.465993 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.465967    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:32.468279 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.468260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.468375 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.468351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:32.472495 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.472418    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"f2eb497638654ab51d85f8c48e93d1a5","Type":"ContainerStarted","Data":"0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939"}
Nov 03 17:44:32.473585 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.473566    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"acca05a04d25dcd6bb0ab5ae22e6769a","Type":"ContainerStarted","Data":"989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea"}
Nov 03 17:44:32.484951 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.484931    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:44:32.498930 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.498909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-env-overrides\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499028 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.498937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9984e6b-2191-4627-96ac-9d0b53762a46-hosts-file\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.499028 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.498957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-bin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499028 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.498979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.499150 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.499150 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-kubelet\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499150 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-node-log\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499283 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-config\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499283 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9984e6b-2191-4627-96ac-9d0b53762a46-tmp-dir\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.499283 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499376 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-conf-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499376 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-modprobe-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.499514 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzbb\" (UniqueName: \"kubernetes.io/projected/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kube-api-access-jwzbb\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.499514 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dnr\" (UniqueName: \"kubernetes.io/projected/5a1a652a-2b2f-412a-bf99-3fab0319dd38-kube-api-access-k9dnr\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.499514 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-host\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.499514 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-system-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499514 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-multus\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-etc-kubernetes\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-host\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.499693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-tmp\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.499693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-systemd-units\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-log-socket\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcj2\" (UniqueName: \"kubernetes.io/projected/1ce9cf1a-209e-489c-bae4-552a734e3e91-kube-api-access-tdcj2\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.499898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.499898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.499898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-tuned\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.499898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-slash\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.499898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-netd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.500222 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.500222 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-hostroot\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.500222 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.499965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-multus-certs\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.500222 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-lib-modules\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.500531 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-ovn\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.500588 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.500588 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.500588 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.500748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjxd\" (UniqueName: \"kubernetes.io/projected/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-kube-api-access-cxjxd\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.500748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-device-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.500748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e77045a-8555-4a49-9ccd-47a14931763b-host-slash\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.500748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx8r\" (UniqueName: \"kubernetes.io/projected/7e77045a-8555-4a49-9ccd-47a14931763b-kube-api-access-xnx8r\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.500748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovn-node-metrics-cert\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.500748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50141ec0-76c0-453c-9ecc-e5c7c227dad2-agent-certs\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50141ec0-76c0-453c-9ecc-e5c7c227dad2-konnectivity-ca\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmwq\" (UniqueName: \"kubernetes.io/projected/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-kube-api-access-2nmwq\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-cni-binary-copy\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-k8s-cni-cncf-io\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-daemon-config\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.500970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-conf\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-script-lib\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-netns\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.501079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e77045a-8555-4a49-9ccd-47a14931763b-iptables-alerter-script\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-var-lib-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-serviceca\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysconfig\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-kubernetes\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-system-cni-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-cnibin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cnibin\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-systemd\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-dbus\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-etc-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-bin\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-socket-dir-parent\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-kubelet-config\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.501663 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gxq\" (UniqueName: \"kubernetes.io/projected/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-kube-api-access-82gxq\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-kubelet\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-sys\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-netns\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vks\" (UniqueName: \"kubernetes.io/projected/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-kube-api-access-z2vks\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4jh\" (UniqueName: \"kubernetes.io/projected/b9984e6b-2191-4627-96ac-9d0b53762a46-kube-api-access-4c4jh\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-run\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-var-lib-kubelet\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-systemd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.502492 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.501712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.504686 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.504668    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.504770 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.504762    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.504870 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.504856    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:44:32.504919 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.504908    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.504951 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.504920    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:44:32.504994 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.504956    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.505805 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.505756    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:44:32.505943 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.505931    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j4vc9\""
Nov 03 17:44:32.506003 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.505974    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mk8dz\""
Nov 03 17:44:32.507493 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.507476    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z56rs\""
Nov 03 17:44:32.507908 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.507893    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.508148 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.508134    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.508186 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.508136    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:44:32.508372 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.508357    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:44:32.559771 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.559643    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m6lsw\""
Nov 03 17:44:32.559898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.559800    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8w5gn\""
Nov 03 17:44:32.559898 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.559862    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bmd8j\""
Nov 03 17:44:32.560109 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560091    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:44:32.560194 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560125    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7l2n\""
Nov 03 17:44:32.560301 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560286    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:44:32.560388 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560336    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8lnxv\""
Nov 03 17:44:32.560464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560436    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:44:32.560532 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560469    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5hxz7\""
Nov 03 17:44:32.561930 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560613    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:44:32.561930 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.560689    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6l8nz"
Nov 03 17:44:32.562098 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.561977    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.562178 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562148    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.562178 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562163    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:44:32.562292 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562210    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.562292 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562234    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:44:32.562292 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562277    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:44:32.562292 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562276    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.562497 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562328    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.562497 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562342    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:44:32.562497 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562429    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:44:32.562638 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562625    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.562638 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.562632    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:44:32.563612 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.563598    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:44:32.602592 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-slash\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-netd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-hostroot\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-multus-certs\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-lib-modules\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602696    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-netd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602706    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-slash\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.602752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602706    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602753    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-multus-certs\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-ovn\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602798    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-ovn\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602757    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-hostroot\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602825    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-lib-modules\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjxd\" (UniqueName: \"kubernetes.io/projected/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-kube-api-access-cxjxd\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.602990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-device-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e77045a-8555-4a49-9ccd-47a14931763b-host-slash\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603014    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx8r\" (UniqueName: \"kubernetes.io/projected/7e77045a-8555-4a49-9ccd-47a14931763b-kube-api-access-xnx8r\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603069    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-device-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.603134 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603094    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e77045a-8555-4a49-9ccd-47a14931763b-host-slash\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovn-node-metrics-cert\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50141ec0-76c0-453c-9ecc-e5c7c227dad2-agent-certs\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50141ec0-76c0-453c-9ecc-e5c7c227dad2-konnectivity-ca\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmwq\" (UniqueName: \"kubernetes.io/projected/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-kube-api-access-2nmwq\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-cni-binary-copy\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-k8s-cni-cncf-io\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-daemon-config\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-conf\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603370    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-k8s-cni-cncf-io\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-script-lib\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-netns\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e77045a-8555-4a49-9ccd-47a14931763b-iptables-alerter-script\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-var-lib-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-serviceca\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.603957 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysconfig\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603535    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-kubernetes\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-system-cni-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603620    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-conf\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603641    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-var-lib-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-cnibin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603682    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-netns\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cnibin\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603733    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-system-cni-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-systemd\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-dbus\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-etc-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-bin\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-socket-dir-parent\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.604925 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-kubelet-config\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.603979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gxq\" (UniqueName: \"kubernetes.io/projected/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-kube-api-access-82gxq\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-kubelet\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-sys\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604095    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50141ec0-76c0-453c-9ecc-e5c7c227dad2-konnectivity-ca\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604110    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-netns\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604142    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-daemon-config\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604159    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-script-lib\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604172    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-netns\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604182    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vks\" (UniqueName: \"kubernetes.io/projected/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-kube-api-access-z2vks\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4jh\" (UniqueName: \"kubernetes.io/projected/b9984e6b-2191-4627-96ac-9d0b53762a46-kube-api-access-4c4jh\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604234    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-cnibin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604275    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-serviceca\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.605595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-run\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604302    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e77045a-8555-4a49-9ccd-47a14931763b-iptables-alerter-script\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-var-lib-kubelet\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.604335    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604348    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysconfig\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-systemd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604391    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-systemd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.604408    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:33.10437355 +0000 UTC m=+3.233370136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604421    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-kubernetes\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604442    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cnibin\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604486    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604497    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-systemd\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604502    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-etc-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-env-overrides\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9984e6b-2191-4627-96ac-9d0b53762a46-hosts-file\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604550    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-socket-dir-parent\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.606401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604555    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-kubelet-config\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-run\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604588    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604595    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-cni-binary-copy\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604598    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9984e6b-2191-4627-96ac-9d0b53762a46-hosts-file\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604613    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-var-lib-kubelet\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-bin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.604629    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604677    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-bin\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604772    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.604789    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:33.104770143 +0000 UTC m=+3.233766723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-kubelet\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604864    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-sys\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604889    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-dbus\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-node-log\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.607278 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-config\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604936    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-bin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604956    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-node-log\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9984e6b-2191-4627-96ac-9d0b53762a46-tmp-dir\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.604973    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-kubelet\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-conf-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-modprobe-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605144    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605199    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605215    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-kubelet\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605235    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-modprobe-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605234    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-conf-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzbb\" (UniqueName: \"kubernetes.io/projected/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kube-api-access-jwzbb\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dnr\" (UniqueName: \"kubernetes.io/projected/5a1a652a-2b2f-412a-bf99-3fab0319dd38-kube-api-access-k9dnr\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-host\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-system-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-multus\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-etc-kubernetes\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605550    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-host\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605592    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-multus\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605592    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-etc-kubernetes\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605612    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-system-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-host\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-tmp\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-systemd-units\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-log-socket\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605735    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-host\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcj2\" (UniqueName: \"kubernetes.io/projected/1ce9cf1a-209e-489c-bae4-552a734e3e91-kube-api-access-tdcj2\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605760    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-systemd-units\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605780    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-log-socket\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605829    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-tuned\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.608920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.605903    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.606008    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-env-overrides\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.606240    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9984e6b-2191-4627-96ac-9d0b53762a46-tmp-dir\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.606241    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-config\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.606421    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.608024    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-tmp\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.608060    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovn-node-metrics-cert\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.608136    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50141ec0-76c0-453c-9ecc-e5c7c227dad2-agent-certs\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.609422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.608571    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-tuned\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.677868 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.677834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjxd\" (UniqueName: \"kubernetes.io/projected/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-kube-api-access-cxjxd\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.677999 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.677924    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx8r\" (UniqueName: \"kubernetes.io/projected/7e77045a-8555-4a49-9ccd-47a14931763b-kube-api-access-xnx8r\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.677999 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.677948    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vks\" (UniqueName: \"kubernetes.io/projected/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-kube-api-access-z2vks\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.684917 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.684895    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4jh\" (UniqueName: \"kubernetes.io/projected/b9984e6b-2191-4627-96ac-9d0b53762a46-kube-api-access-4c4jh\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.685052 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.684962    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gxq\" (UniqueName: \"kubernetes.io/projected/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-kube-api-access-82gxq\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.685854 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.685836    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcj2\" (UniqueName: \"kubernetes.io/projected/1ce9cf1a-209e-489c-bae4-552a734e3e91-kube-api-access-tdcj2\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.685928 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.685885    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzbb\" (UniqueName: \"kubernetes.io/projected/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kube-api-access-jwzbb\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.688787 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.688755    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:32.688787 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.688789    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:32.688949 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.688816    2581 projected.go:196] Error preparing data for projected volume kube-api-access-b68wl for pod openshift-network-diagnostics/network-check-target-jrt48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:32.688949 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:32.688883    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl podName:8d645b31-5684-42a1-b305-59fe985149c4 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:33.188861372 +0000 UTC m=+3.317857946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b68wl" (UniqueName: "kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl") pod "network-check-target-jrt48" (UID: "8d645b31-5684-42a1-b305-59fe985149c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:32.690769 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.690743    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dnr\" (UniqueName: \"kubernetes.io/projected/5a1a652a-2b2f-412a-bf99-3fab0319dd38-kube-api-access-k9dnr\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.690928 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.690911    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmwq\" (UniqueName: \"kubernetes.io/projected/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-kube-api-access-2nmwq\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:32.745531 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.745511    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:44:32.751026 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.751003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:44:32.761156 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.761133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qkcgs"
Nov 03 17:44:32.761743 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.761713    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9984e6b_2191_4627_96ac_9d0b53762a46.slice/crio-d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d WatchSource:0}: Error finding container d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d: Status 404 returned error can't find the container with id d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d
Nov 03 17:44:32.771219 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.771197    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce9cf1a_209e_489c_bae4_552a734e3e91.slice/crio-5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7 WatchSource:0}: Error finding container 5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7: Status 404 returned error can't find the container with id 5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7
Nov 03 17:44:32.771790 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.771765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:44:32.780195 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.780172    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b9dedb_2e8f_47d8_9a9f_3398edae5365.slice/crio-df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834 WatchSource:0}: Error finding container df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834: Status 404 returned error can't find the container with id df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834
Nov 03 17:44:32.784496 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.784481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:44:32.787826 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.787809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:44:32.789303 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.789290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:44:32.794569 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.794545    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e77045a_8555_4a49_9ccd_47a14931763b.slice/crio-7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048 WatchSource:0}: Error finding container 7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048: Status 404 returned error can't find the container with id 7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048
Nov 03 17:44:32.798011 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.797987    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1a652a_2b2f_412a_bf99_3fab0319dd38.slice/crio-bbfa63624ceea5bfef98b4ee88e60e216cb96214bb1111b87f1f3cd2c8358d77 WatchSource:0}: Error finding container bbfa63624ceea5bfef98b4ee88e60e216cb96214bb1111b87f1f3cd2c8358d77: Status 404 returned error can't find the container with id bbfa63624ceea5bfef98b4ee88e60e216cb96214bb1111b87f1f3cd2c8358d77
Nov 03 17:44:32.799999 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.799976    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcba5ac_7a87_4aa9_b3e3_a474946ba957.slice/crio-469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b WatchSource:0}: Error finding container 469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b: Status 404 returned error can't find the container with id 469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b
Nov 03 17:44:32.803170 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.803153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:32.806349 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.806333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:44:32.812872 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:32.812851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:32.816163 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.816146    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ef9fc0_8b2b_440a_bdd1_5684a5bc573d.slice/crio-5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce WatchSource:0}: Error finding container 5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce: Status 404 returned error can't find the container with id 5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce
Nov 03 17:44:32.822124 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:44:32.822104    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cd964f_0f58_48bb_a2b5_81b1d0f355b2.slice/crio-78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e WatchSource:0}: Error finding container 78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e: Status 404 returned error can't find the container with id 78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e
Nov 03 17:44:33.110220 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.110137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:33.110386 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.110221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:33.110386 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.110307    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:33.110386 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.110366    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:34.110351952 +0000 UTC m=+4.239348514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:33.110563 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.110385    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:33.110563 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.110450    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:34.110432881 +0000 UTC m=+4.239429446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:33.211625 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.211585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:33.211840 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.211817    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:33.211840 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.211840    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:33.211965 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.211852    2581 projected.go:196] Error preparing data for projected volume kube-api-access-b68wl for pod openshift-network-diagnostics/network-check-target-jrt48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:33.211965 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:33.211914    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl podName:8d645b31-5684-42a1-b305-59fe985149c4 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:34.211894779 +0000 UTC m=+4.340891348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b68wl" (UniqueName: "kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl") pod "network-check-target-jrt48" (UID: "8d645b31-5684-42a1-b305-59fe985149c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:33.477583 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.477488    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerStarted","Data":"469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b"}
Nov 03 17:44:33.479277 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.479248    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8nvv" event={"ID":"5a1a652a-2b2f-412a-bf99-3fab0319dd38","Type":"ContainerStarted","Data":"bbfa63624ceea5bfef98b4ee88e60e216cb96214bb1111b87f1f3cd2c8358d77"}
Nov 03 17:44:33.480487 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.480461    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hn245" event={"ID":"7e77045a-8555-4a49-9ccd-47a14931763b","Type":"ContainerStarted","Data":"7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048"}
Nov 03 17:44:33.481858 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.481826    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4" event={"ID":"03b9dedb-2e8f-47d8-9a9f-3398edae5365","Type":"ContainerStarted","Data":"df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834"}
Nov 03 17:44:33.482993 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.482968    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mnmh" event={"ID":"b9984e6b-2191-4627-96ac-9d0b53762a46","Type":"ContainerStarted","Data":"d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d"}
Nov 03 17:44:33.484248 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.484227    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e"}
Nov 03 17:44:33.485513 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.485490    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8z24k" event={"ID":"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d","Type":"ContainerStarted","Data":"5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce"}
Nov 03 17:44:33.486779 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.486754    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2vd6" event={"ID":"50141ec0-76c0-453c-9ecc-e5c7c227dad2","Type":"ContainerStarted","Data":"d6c626d2b96509e6222701aac7c8d0e2d1d28e8d572da2f403b8a3601653c26a"}
Nov 03 17:44:33.487896 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.487876    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qkcgs" event={"ID":"1ce9cf1a-209e-489c-bae4-552a734e3e91","Type":"ContainerStarted","Data":"5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7"}
Nov 03 17:44:33.564125 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.564049    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:32 +0000 UTC" deadline="2028-07-02 06:42:50.35859095 +0000 UTC"
Nov 03 17:44:33.564125 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:33.564118    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14556h58m16.794475886s"
Nov 03 17:44:34.119425 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.119368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:34.119633 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.119448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:34.119725 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.119707    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:34.119833 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.119819    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:36.119795957 +0000 UTC m=+6.248792535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:34.119935 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.119920    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:34.120005 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.119973    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:36.119958129 +0000 UTC m=+6.248954693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:34.220334 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.219947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:34.220334 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.220203    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:34.220334 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.220227    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:34.220334 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.220241    2581 projected.go:196] Error preparing data for projected volume kube-api-access-b68wl for pod openshift-network-diagnostics/network-check-target-jrt48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:34.220334 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.220303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl podName:8d645b31-5684-42a1-b305-59fe985149c4 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:36.22028373 +0000 UTC m=+6.349280313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b68wl" (UniqueName: "kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl") pod "network-check-target-jrt48" (UID: "8d645b31-5684-42a1-b305-59fe985149c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:34.471253 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.471179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:34.471420 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.471328    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:34.471463 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.471420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:34.471526 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.471507    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:34.471589 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.471573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:34.471672 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:34.471652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:34.565308 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.565262    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:32 +0000 UTC" deadline="2028-08-17 17:28:23.819649718 +0000 UTC"
Nov 03 17:44:34.565308 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:34.565305    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15671h43m49.254348802s"
Nov 03 17:44:35.508121 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:35.507871    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"f2eb497638654ab51d85f8c48e93d1a5","Type":"ContainerStarted","Data":"180c24f7be4aeea835a131d6a3fcc19e74743167c5ad5ad238e0ba517be31e8d"}
Nov 03 17:44:36.135265 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.135225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:36.135708 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.135283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:36.135708 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.135440    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:36.135708 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.135498    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:40.135481702 +0000 UTC m=+10.264478273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:36.135708 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.135564    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:36.135708 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.135597    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:40.135586224 +0000 UTC m=+10.264582783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:36.235922 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.235887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:36.236097 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.236077    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:36.236163 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.236102    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:36.236163 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.236115    2581 projected.go:196] Error preparing data for projected volume kube-api-access-b68wl for pod openshift-network-diagnostics/network-check-target-jrt48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:36.236275 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.236172    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl podName:8d645b31-5684-42a1-b305-59fe985149c4 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:40.236155123 +0000 UTC m=+10.365151695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b68wl" (UniqueName: "kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl") pod "network-check-target-jrt48" (UID: "8d645b31-5684-42a1-b305-59fe985149c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:36.470545 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.470077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:36.470545 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.470196    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:36.470545 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.470271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:36.470545 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.470374    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:36.470545 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.470428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:36.470545 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:36.470491    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:36.526427 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.525592    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2eb497638654ab51d85f8c48e93d1a5" containerID="180c24f7be4aeea835a131d6a3fcc19e74743167c5ad5ad238e0ba517be31e8d" exitCode=0
Nov 03 17:44:36.526427 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.525677    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"f2eb497638654ab51d85f8c48e93d1a5","Type":"ContainerDied","Data":"180c24f7be4aeea835a131d6a3fcc19e74743167c5ad5ad238e0ba517be31e8d"}
Nov 03 17:44:36.530843 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:36.530811    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerStarted","Data":"06045d4c5f169da6d2b4933fbf2756d048d495713f2b6fda7eb56a668ab7bd69"}
Nov 03 17:44:37.538988 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:37.538952    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"f2eb497638654ab51d85f8c48e93d1a5","Type":"ContainerStarted","Data":"e1b4ae0eef2c79d1ace05955599451ce8edeaf55bd361f18c32bf587594fc21b"}
Nov 03 17:44:37.544314 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:37.544289    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="06045d4c5f169da6d2b4933fbf2756d048d495713f2b6fda7eb56a668ab7bd69" exitCode=0
Nov 03 17:44:37.544425 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:37.544332    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerDied","Data":"06045d4c5f169da6d2b4933fbf2756d048d495713f2b6fda7eb56a668ab7bd69"}
Nov 03 17:44:37.795058 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:37.794966    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal" podStartSLOduration=5.794945913 podStartE2EDuration="5.794945913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:37.663329776 +0000 UTC m=+7.792326358" watchObservedRunningTime="2026-11-03 17:44:37.794945913 +0000 UTC m=+7.923942495"
Nov 03 17:44:38.470816 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:38.470738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:38.470970 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:38.470884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:38.470970 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:38.470964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:38.471744 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:38.471040    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:38.471744 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:38.471101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:38.471744 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:38.471187    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:38.550947 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:38.550903    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"acca05a04d25dcd6bb0ab5ae22e6769a","Type":"ContainerStarted","Data":"379bc65cd49430b284cbe11779f1a4409b30ee442272a39bca2cccf85b867595"}
Nov 03 17:44:38.555552 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:38.555423    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2vd6" event={"ID":"50141ec0-76c0-453c-9ecc-e5c7c227dad2","Type":"ContainerStarted","Data":"9fa943d737f610d17bf912eb89be25f99793a34066550b649d601fb9c07c181d"}
Nov 03 17:44:38.804099 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:38.803237    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal" podStartSLOduration=6.803221188 podStartE2EDuration="6.803221188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:38.674882636 +0000 UTC m=+8.803879215" watchObservedRunningTime="2026-11-03 17:44:38.803221188 +0000 UTC m=+8.932217768"
Nov 03 17:44:39.561661 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:39.561172    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8z24k" event={"ID":"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d","Type":"ContainerStarted","Data":"6bc42ac382808d7f630d34291b031d732604c617ae094198d39201e1b791483e"}
Nov 03 17:44:39.683095 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:39.683021    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j2vd6" podStartSLOduration=3.461350036 podStartE2EDuration="8.683001701s" totalImagesPullingTime="5.221651665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:38.804072458 +0000 UTC m=+8.933069038" watchObservedRunningTime="2026-11-03 17:44:39.683001701 +0000 UTC m=+9.811998281"
Nov 03 17:44:39.683263 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:39.683148    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8z24k" podStartSLOduration=2.945339047 podStartE2EDuration="8.683141328s" totalImagesPullingTime="5.737802281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:39.682569072 +0000 UTC m=+9.811565651" watchObservedRunningTime="2026-11-03 17:44:39.683141328 +0000 UTC m=+9.812137909"
Nov 03 17:44:40.170784 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.170341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:40.170784 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.170398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:40.170784 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.170572    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:40.170784 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.170634    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:48.170617248 +0000 UTC m=+18.299613828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:40.170784 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.170702    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:40.170784 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.170750    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:48.170737568 +0000 UTC m=+18.299734124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:40.271695 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.271658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:40.271893 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.271845    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:40.271893 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.271866    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:40.271893 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.271879    2581 projected.go:196] Error preparing data for projected volume kube-api-access-b68wl for pod openshift-network-diagnostics/network-check-target-jrt48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:40.272051 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.271938    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl podName:8d645b31-5684-42a1-b305-59fe985149c4 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:48.271919044 +0000 UTC m=+18.400915623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b68wl" (UniqueName: "kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl") pod "network-check-target-jrt48" (UID: "8d645b31-5684-42a1-b305-59fe985149c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:40.469964 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.468846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:40.469964 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.468852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:40.469964 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.469592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:40.469964 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.469702    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:40.469964 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.469799    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:40.469964 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:40.469885    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:40.565456 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:40.565424    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4" event={"ID":"03b9dedb-2e8f-47d8-9a9f-3398edae5365","Type":"ContainerStarted","Data":"9039d8db5856e70ea1cb3147d338b92a887c662bf22f8d862ec0e55a9142e030"}
Nov 03 17:44:41.569144 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:41.568929    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qkcgs" event={"ID":"1ce9cf1a-209e-489c-bae4-552a734e3e91","Type":"ContainerStarted","Data":"3f4aa4f19c64d77c556d046667febe2b760c25e34ce40f08a4ce14316555f50d"}
Nov 03 17:44:41.842233 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:41.842185    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:44:42.468506 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.468468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:42.468782 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:42.468600    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:42.468782 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.468620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:42.468782 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.468646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:42.468782 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:42.468753    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:42.469110 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:42.468834    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:42.493696 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.493588    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:44:41.842206172Z","UUID":"8f574141-a823-49ee-91ea-420eef7ed3d9","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:44:42.495183 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.495158    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:44:42.495318 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.495192    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:44:42.573461 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.573425    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mnmh" event={"ID":"b9984e6b-2191-4627-96ac-9d0b53762a46","Type":"ContainerStarted","Data":"3ef861ce43fbb2510fee1ba90a1b7ae22ef0acef3ec4557920ab72acd7223830"}
Nov 03 17:44:42.575525 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.575489    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="02fe4bab5770d0aea6c47b8829b991a82885a7d92aba722a359fc5b9b707a1b3" exitCode=0
Nov 03 17:44:42.575649 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.575581    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerDied","Data":"02fe4bab5770d0aea6c47b8829b991a82885a7d92aba722a359fc5b9b707a1b3"}
Nov 03 17:44:42.578535 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.578509    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hn245" event={"ID":"7e77045a-8555-4a49-9ccd-47a14931763b","Type":"ContainerStarted","Data":"4262180b2c2905709ac21cab49b6c5a74e910de616b5a75c35cc917e568af91b"}
Nov 03 17:44:42.580677 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.580653    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4" event={"ID":"03b9dedb-2e8f-47d8-9a9f-3398edae5365","Type":"ContainerStarted","Data":"33acbc134a7e3265f7e7e40e64cadaa1fc4d6e661550be9c92ec92e991e8c051"}
Nov 03 17:44:42.694714 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.694611    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7mnmh" podStartSLOduration=3.9200359000000002 podStartE2EDuration="12.694590468s" totalImagesPullingTime="8.774554568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:42.694339066 +0000 UTC m=+12.823335645" watchObservedRunningTime="2026-11-03 17:44:42.694590468 +0000 UTC m=+12.823587047"
Nov 03 17:44:42.695273 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.695234    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qkcgs" podStartSLOduration=4.421615484 podStartE2EDuration="12.695222553s" totalImagesPullingTime="8.273607069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:41.696401072 +0000 UTC m=+11.825397629" watchObservedRunningTime="2026-11-03 17:44:42.695222553 +0000 UTC m=+12.824219132"
Nov 03 17:44:42.965235 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:42.965139    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hn245" podStartSLOduration=3.866046454 podStartE2EDuration="11.965123931s" totalImagesPullingTime="8.099077477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:42.964754606 +0000 UTC m=+13.093751185" watchObservedRunningTime="2026-11-03 17:44:42.965123931 +0000 UTC m=+13.094120509"
Nov 03 17:44:43.198783 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:43.198759    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:43.199298 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:43.199278    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:44:43.586355 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:43.586317    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4" event={"ID":"03b9dedb-2e8f-47d8-9a9f-3398edae5365","Type":"ContainerStarted","Data":"26a25410387010673c9a13065c351032b42757485f78fd6f32d0a6e2fbeae419"}
Nov 03 17:44:43.587825 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:43.587794    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8nvv" event={"ID":"5a1a652a-2b2f-412a-bf99-3fab0319dd38","Type":"ContainerStarted","Data":"defe6273ae1b7adbf317210c5495f0b1b11a23f000c02196bd43afa0b0243b8e"}
Nov 03 17:44:43.711176 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:43.710921    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4" podStartSLOduration=2.499675092 podStartE2EDuration="12.71090664s" totalImagesPullingTime="10.211231548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:43.710874643 +0000 UTC m=+13.839871222" watchObservedRunningTime="2026-11-03 17:44:43.71090664 +0000 UTC m=+13.839903216"
Nov 03 17:44:44.468897 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:44.468814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:44.469067 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:44.468830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:44.469067 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:44.468917    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:44.469067 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:44.468949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:44.469067 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:44.469035    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:44.469257 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:44.469102    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:44.592805 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:44.592774    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="7d96b759f0ce1ab6ddec3bdf3894899774ccfa8b06314eadbf412bb00dfe190d" exitCode=0
Nov 03 17:44:44.593229 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:44.592872    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerDied","Data":"7d96b759f0ce1ab6ddec3bdf3894899774ccfa8b06314eadbf412bb00dfe190d"}
Nov 03 17:44:44.724697 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:44.724593    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m8nvv" podStartSLOduration=3.387701882 podStartE2EDuration="13.72457652s" totalImagesPullingTime="10.336874638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:43.848926083 +0000 UTC m=+13.977922662" watchObservedRunningTime="2026-11-03 17:44:44.72457652 +0000 UTC m=+14.853573090"
Nov 03 17:44:46.468217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:46.468179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:46.468674 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:46.468179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:46.468674 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:46.468312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:46.468674 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:46.468179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:46.468674 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:46.468385    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:46.468674 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:46.468436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:46.600775 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:46.600744    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="88fe810013ea6547b61320f27dbf0c47dbec63da3fc58d3e425c7d44aa5b1e50" exitCode=0
Nov 03 17:44:46.600959 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:46.600791    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerDied","Data":"88fe810013ea6547b61320f27dbf0c47dbec63da3fc58d3e425c7d44aa5b1e50"}
Nov 03 17:44:48.236080 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:48.236037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:48.236602 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:48.236101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:48.236602 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.236169    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:48.236602 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.236246    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.236224993 +0000 UTC m=+34.365221551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:44:48.236602 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.236278    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:48.236602 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.236339    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.236319857 +0000 UTC m=+34.365316416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:44:48.337402 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:48.337362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:48.337583 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.337526    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:44:48.337583 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.337548    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:44:48.337583 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.337561    2581 projected.go:196] Error preparing data for projected volume kube-api-access-b68wl for pod openshift-network-diagnostics/network-check-target-jrt48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:48.337748 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.337624    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl podName:8d645b31-5684-42a1-b305-59fe985149c4 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.337607094 +0000 UTC m=+34.466603652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b68wl" (UniqueName: "kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl") pod "network-check-target-jrt48" (UID: "8d645b31-5684-42a1-b305-59fe985149c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:44:48.467864 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:48.467831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:48.468020 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.467963    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:48.468181 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:48.468160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:48.468324 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.468254    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:48.468461 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:48.468444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:48.468549 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:48.468532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:49.610247 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:49.610211    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="96f21f245e5a3b2e7f932bd2d2f46fba1ebc0c5bb18f48b64ef5deac51fdb55a" exitCode=0
Nov 03 17:44:49.610744 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:49.610264    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerDied","Data":"96f21f245e5a3b2e7f932bd2d2f46fba1ebc0c5bb18f48b64ef5deac51fdb55a"}
Nov 03 17:44:50.468704 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.468524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:50.468876 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.468601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:50.468876 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:50.468797    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:50.468876 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.468629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:50.468876 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:50.468852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:50.469049 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:50.468911    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:50.614809 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.614777    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"0493a6994226c49de459531770acf79d2669e350af01e965826b8216418880d6"}
Nov 03 17:44:50.615510 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.614872    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"7eb8aa7795df5a55c797c32bbbf80d638a12a526e4b1765c8946bba16ec32649"}
Nov 03 17:44:50.615510 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.614911    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"d2756e7dbf65105b0da9944bc8403e3d0544d75c0b62e66c5d62dcaaca7893b7"}
Nov 03 17:44:50.615510 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.614963    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"b181b63f36b0d45a608a31dfa6d10ddd5e63db1c814f2157a0bcb494d14612ad"}
Nov 03 17:44:50.615510 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.615007    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"a447e116bc55f768d3be0153d58fdfc77054aea860d60157ef64e975480e8f34"}
Nov 03 17:44:50.615510 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.615044    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"f29e7650fb2637c4e5f622beb9ecd972691728482670ede2ddcb59a7223903f6"}
Nov 03 17:44:50.620204 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.620179    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="2c72b5889ea8628b9894725d2ed18cb6c95f13cce8906cb26310566e88164368" exitCode=0
Nov 03 17:44:50.620307 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:50.620220    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerDied","Data":"2c72b5889ea8628b9894725d2ed18cb6c95f13cce8906cb26310566e88164368"}
Nov 03 17:44:51.624756 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:51.624709    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" event={"ID":"5bcba5ac-7a87-4aa9-b3e3-a474946ba957","Type":"ContainerStarted","Data":"e4bbeb28dd5a1b1287acb702be2801a5d6dab89ea713a87f0a07a4aa197c8555"}
Nov 03 17:44:51.753189 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:51.753138    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nmrf9" podStartSLOduration=8.05980885 podStartE2EDuration="20.75312341s" totalImagesPullingTime="12.69331456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:51.752941785 +0000 UTC m=+21.881938363" watchObservedRunningTime="2026-11-03 17:44:51.75312341 +0000 UTC m=+21.882119988"
Nov 03 17:44:52.468373 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:52.468282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:52.468524 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:52.468385    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:52.468524 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:52.468407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:52.468524 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:52.468282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:52.468524 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:52.468503    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:52.468654 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:52.468625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:52.630511 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:52.630474    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"0bf299ca9db52d4a82894c6a793f3128248f8dc308077c114615be4edb155a11"}
Nov 03 17:44:54.468320 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.468204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:54.468320 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.468210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:54.468844 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:54.468326    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:54.468844 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.468209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:54.468844 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:54.468416    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:54.468844 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:54.468471    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:54.638460 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.638424    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" event={"ID":"07cd964f-0f58-48bb-a2b5-81b1d0f355b2","Type":"ContainerStarted","Data":"1b00479c55d023323673250737e428aca4101ef6b154924510adcdf92cdf37c1"}
Nov 03 17:44:54.638602 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.638591    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:54.638711 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.638700    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:54.638782 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.638760    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:54.653641 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.653618    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:54.658124 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.658103    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:44:54.934371 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:54.934283    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8" podStartSLOduration=6.943303298 podStartE2EDuration="23.934270449s" totalImagesPullingTime="16.990967151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:54.934006725 +0000 UTC m=+25.063003304" watchObservedRunningTime="2026-11-03 17:44:54.934270449 +0000 UTC m=+25.063267046"
Nov 03 17:44:56.468157 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.468122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:56.468157 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.468141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:56.468643 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:56.468222    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:56.468643 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:56.468354    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:56.468643 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.468422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:56.468643 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:56.468501    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:56.943315 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.943244    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zc2rm"]
Nov 03 17:44:56.943446 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.943371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:56.943513 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:56.943483    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:44:56.945343 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.945320    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jrt48"]
Nov 03 17:44:56.945438 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.945402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:56.945495 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:56.945480    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:56.948343 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.948320    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8b682"]
Nov 03 17:44:56.948429 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:56.948413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:56.948531 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:56.948514    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:58.468502 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:58.468468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:44:58.468502 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:58.468480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:44:58.469099 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:58.468610    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:44:58.469099 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:44:58.468654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:44:58.469099 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:58.468754    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:44:58.469099 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:44:58.468803    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:45:00.468501 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:00.468472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:00.468971 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:00.468550    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:45:00.468971 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:00.468656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:00.468971 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:00.468791    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:45:00.468971 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:00.468800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:00.468971 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:00.468884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:45:01.170229 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:01.170199    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:45:01.170399 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:01.170337    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:45:01.170807 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:01.170784    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:45:02.468032 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:02.467788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:02.468421 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:02.467799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:02.468421 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:02.468153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b682" podUID="d5703035-ada3-4cfb-b77b-4a8e9bcbe063"
Nov 03 17:45:02.468421 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:02.467819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:02.468421 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:02.468178    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zc2rm" podUID="9807c6c6-ef8b-4769-bf2e-97b5c5a23604"
Nov 03 17:45:02.468421 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:02.468251    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrt48" podUID="8d645b31-5684-42a1-b305-59fe985149c4"
Nov 03 17:45:03.132641 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.132616    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeReady"
Nov 03 17:45:03.132824 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.132747    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 17:45:03.253901 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.253871    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"]
Nov 03 17:45:03.329615 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.329585    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"]
Nov 03 17:45:03.329778 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.329705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:03.342497 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.342473    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65f6r"]
Nov 03 17:45:03.342603 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.342544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.356766 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.356741    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dsfq"]
Nov 03 17:45:03.356977 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.356820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:03.383245 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.383190    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txjqv"]
Nov 03 17:45:03.383352 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.383311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:03.397337 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.397315    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92"]
Nov 03 17:45:03.397450 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.397434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:03.411509 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.411476    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"]
Nov 03 17:45:03.411651 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.411620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.413641 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.413621    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.435689 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.435667    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"]
Nov 03 17:45:03.435801 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.435689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.441403 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.441381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-config\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.441504 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.441483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjm7\" (UniqueName: \"kubernetes.io/projected/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-kube-api-access-jpjm7\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.441565 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.441537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbjl\" (UniqueName: \"kubernetes.io/projected/9392d9f2-6a5c-4b6a-b048-a20227a79836-kube-api-access-5qbjl\") pod \"kas-connection-checker-685895c7c9-65f6r\" (UID: \"9392d9f2-6a5c-4b6a-b048-a20227a79836\") " pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:03.441632 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.441613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2rc\" (UniqueName: \"kubernetes.io/projected/4e4803ee-0fee-44d8-a5d9-7931656d6e7c-kube-api-access-rk2rc\") pod \"volume-data-source-validator-58c9f5f46d-h8s5c\" (UID: \"4e4803ee-0fee-44d8-a5d9-7931656d6e7c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:03.441675 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.441659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-serving-cert\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.442518 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.442501    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 17:45:03.442758 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.442743    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6r97c\""
Nov 03 17:45:03.442805 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.442743    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kf4nv\""
Nov 03 17:45:03.459224 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.459199    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"]
Nov 03 17:45:03.459361 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.459325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:03.473326 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.473308    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"]
Nov 03 17:45:03.473617 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.473420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.487445 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.487429    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c466dc58-kmk8g"]
Nov 03 17:45:03.487566 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.487551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.489973 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.489954    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.490051 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.490008    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.490084 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.490071    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.490116 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.490083    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 17:45:03.490151 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.490143    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.490217 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.490194    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.491836 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.491819    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 17:45:03.501787 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.501769    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"]
Nov 03 17:45:03.501884 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.501870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.509156 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.509138    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 17:45:03.515863 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.515844    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"]
Nov 03 17:45:03.515964 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.515949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:03.536041 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.536013    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.536171 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.536022    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6nngz"]
Nov 03 17:45:03.541995 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.541975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjm7\" (UniqueName: \"kubernetes.io/projected/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-kube-api-access-jpjm7\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.542090 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/69cbe72d-a018-4a89-8951-a25e50d6d33b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.542090 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.542090 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vm2\" (UniqueName: \"kubernetes.io/projected/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911-kube-api-access-f8vm2\") pod \"kas-connection-checker-685895c7c9-7dsfq\" (UID: \"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911\") " pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:03.542090 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbjl\" (UniqueName: \"kubernetes.io/projected/9392d9f2-6a5c-4b6a-b048-a20227a79836-kube-api-access-5qbjl\") pod \"kas-connection-checker-685895c7c9-65f6r\" (UID: \"9392d9f2-6a5c-4b6a-b048-a20227a79836\") " pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:03.542272 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.542272 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08a559aa-cebd-40a6-9393-3feb287029ee-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:03.542272 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpctd\" (UniqueName: \"kubernetes.io/projected/bae3db47-36e8-4957-9c10-fd361815db36-kube-api-access-zpctd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.542418 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2rc\" (UniqueName: \"kubernetes.io/projected/4e4803ee-0fee-44d8-a5d9-7931656d6e7c-kube-api-access-rk2rc\") pod \"volume-data-source-validator-58c9f5f46d-h8s5c\" (UID: \"4e4803ee-0fee-44d8-a5d9-7931656d6e7c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:03.542418 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-serving-cert\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.542418 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.542558 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.542558 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bae3db47-36e8-4957-9c10-fd361815db36-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.542558 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.542693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:03.542693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7ds\" (UniqueName: \"kubernetes.io/projected/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5-kube-api-access-4m7ds\") pod \"kas-connection-checker-685895c7c9-txjqv\" (UID: \"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5\") " pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:03.542693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-config\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.542826 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.542826 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29nt9\" (UniqueName: \"kubernetes.io/projected/69cbe72d-a018-4a89-8951-a25e50d6d33b-kube-api-access-29nt9\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.542826 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.542967 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bae3db47-36e8-4957-9c10-fd361815db36-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.542967 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542901    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 17:45:03.542967 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542909    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t67n\""
Nov 03 17:45:03.542967 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542970    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.542990    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543006    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7lgsk\""
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543023    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543026    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543049    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 17:45:03.543139 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543104    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gczt4\""
Nov 03 17:45:03.543360 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543236    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-config\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.543587 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.543573    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 17:45:03.546073 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.546056    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-serving-cert\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.564913 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.564893    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b45cc6f6f-ph9d6"]
Nov 03 17:45:03.565020 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.565006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.585555 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.585538    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nkxg8"]
Nov 03 17:45:03.585636 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.585621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.593328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.593312    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.593710 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.593691    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.593809 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.593788    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 17:45:03.593846 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.593824    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.593881 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.593852    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.594048 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594035    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.594087 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594037    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 17:45:03.594087 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594062    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.594087 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594037    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.594087 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594068    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.594202 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594109    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.594202 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594122    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 17:45:03.594202 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594132    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 17:45:03.594202 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.594153    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.595020 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.595005    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 17:45:03.598599 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.598584    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lrmc7"]
Nov 03 17:45:03.598710 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.598698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.607421 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.607403    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbjl\" (UniqueName: \"kubernetes.io/projected/9392d9f2-6a5c-4b6a-b048-a20227a79836-kube-api-access-5qbjl\") pod \"kas-connection-checker-685895c7c9-65f6r\" (UID: \"9392d9f2-6a5c-4b6a-b048-a20227a79836\") " pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:03.607593 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.607575    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjm7\" (UniqueName: \"kubernetes.io/projected/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-kube-api-access-jpjm7\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.607688 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.607675    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2rc\" (UniqueName: \"kubernetes.io/projected/4e4803ee-0fee-44d8-a5d9-7931656d6e7c-kube-api-access-rk2rc\") pod \"volume-data-source-validator-58c9f5f46d-h8s5c\" (UID: \"4e4803ee-0fee-44d8-a5d9-7931656d6e7c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:03.612661 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612643    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"]
Nov 03 17:45:03.612792 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612672    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dsfq"]
Nov 03 17:45:03.612792 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612692    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65f6r"]
Nov 03 17:45:03.612792 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612712    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"]
Nov 03 17:45:03.612792 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612749    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txjqv"]
Nov 03 17:45:03.612792 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612767    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.612985 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612770    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"]
Nov 03 17:45:03.612985 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612927    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"]
Nov 03 17:45:03.612985 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612955    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"]
Nov 03 17:45:03.612985 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612976    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.612999    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613022    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6nngz"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613042    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b45cc6f6f-ph9d6"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613063    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613091    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lrmc7"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613121    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c466dc58-kmk8g"]
Nov 03 17:45:03.613141 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613141    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"]
Nov 03 17:45:03.613332 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.613163    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nkxg8"]
Nov 03 17:45:03.634673 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.634626    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 17:45:03.634673 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.634627    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mpkll\""
Nov 03 17:45:03.634673 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.634630    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 17:45:03.641009 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.640995    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 17:45:03.641503 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.641485    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 17:45:03.641576 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.641526    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 17:45:03.643395 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.643448 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-service-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.643448 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.643448 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08a559aa-cebd-40a6-9393-3feb287029ee-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:03.643693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29nt9\" (UniqueName: \"kubernetes.io/projected/69cbe72d-a018-4a89-8951-a25e50d6d33b-kube-api-access-29nt9\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.643782 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.643782 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bae3db47-36e8-4957-9c10-fd361815db36-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.643782 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:03.643923 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/2fd2bd81-c082-46e4-87c2-226f1acf6c07-kube-api-access-5th4k\") pod \"network-check-source-6f78d59f99-nwhnp\" (UID: \"2fd2bd81-c082-46e4-87c2-226f1acf6c07\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:03.643923 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.643923 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bae3db47-36e8-4957-9c10-fd361815db36-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.643923 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.643852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45jf\" (UniqueName: \"kubernetes.io/projected/16fc5076-624a-47e0-9430-9550882714d8-kube-api-access-p45jf\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.644050 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.643938    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:03.644087 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.644051    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls podName:69cbe72d-a018-4a89-8951-a25e50d6d33b nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.144033203 +0000 UTC m=+34.273029766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8ff8h" (UID: "69cbe72d-a018-4a89-8951-a25e50d6d33b") : secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:03.644087 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.644087 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.644079    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 17:45:03.644290 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.644290 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.644290 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.644167    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert podName:08a559aa-cebd-40a6-9393-3feb287029ee nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.144149994 +0000 UTC m=+34.273146557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkbq7" (UID: "08a559aa-cebd-40a6-9393-3feb287029ee") : secret "networking-console-plugin-cert" not found
Nov 03 17:45:03.644290 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.644290 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644223    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08a559aa-cebd-40a6-9393-3feb287029ee-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:03.644290 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644366    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bae3db47-36e8-4957-9c10-fd361815db36-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef120cf-f11d-435c-96c4-96d73f511afa-serving-cert\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vm2\" (UniqueName: \"kubernetes.io/projected/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911-kube-api-access-f8vm2\") pod \"kas-connection-checker-685895c7c9-7dsfq\" (UID: \"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911\") " pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpctd\" (UniqueName: \"kubernetes.io/projected/bae3db47-36e8-4957-9c10-fd361815db36-kube-api-access-zpctd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-config\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-serving-cert\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.644584 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644763    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7ds\" (UniqueName: \"kubernetes.io/projected/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5-kube-api-access-4m7ds\") pod \"kas-connection-checker-685895c7c9-txjqv\" (UID: \"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5\") " pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbdf\" (UniqueName: \"kubernetes.io/projected/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-kube-api-access-8cbdf\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.644968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-tmp\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-snapshots\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.645173 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/69cbe72d-a018-4a89-8951-a25e50d6d33b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.645752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr87j\" (UniqueName: \"kubernetes.io/projected/9ef120cf-f11d-435c-96c4-96d73f511afa-kube-api-access-tr87j\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.645752 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.645194    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:03.645752 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.645208    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6589fbf699-mdv92: secret "image-registry-tls" not found
Nov 03 17:45:03.645752 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.645248    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls podName:18d852d5-b4c4-4f8f-b2da-bc84b039025e nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.145232673 +0000 UTC m=+34.274229232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls") pod "image-registry-6589fbf699-mdv92" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e") : secret "image-registry-tls" not found
Nov 03 17:45:03.645752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645274    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.645752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645670    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.646112 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.645801    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/69cbe72d-a018-4a89-8951-a25e50d6d33b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.646231 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.646216    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bae3db47-36e8-4957-9c10-fd361815db36-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.646575 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.646549    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.646793 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.646776    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.650626 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.650607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:03.659206 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.659188    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:45:03.659206 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.659201    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xfh9g\""
Nov 03 17:45:03.665294 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.665277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:03.690275 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690255    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.690572 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690524    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.690572 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690531    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:45:03.690572 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690552    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.690572 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690566    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.690572 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690569    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 17:45:03.690866 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690544    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:45:03.690866 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690596    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:45:03.690866 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.690612    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 17:45:03.691353 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.691337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:03.691767 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.691749    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 17:45:03.709908 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.708897    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29nt9\" (UniqueName: \"kubernetes.io/projected/69cbe72d-a018-4a89-8951-a25e50d6d33b-kube-api-access-29nt9\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:03.709908 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.709023    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.709908 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.709101    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:03.709908 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.709303    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpctd\" (UniqueName: \"kubernetes.io/projected/bae3db47-36e8-4957-9c10-fd361815db36-kube-api-access-zpctd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.710374 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.710354    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7ds\" (UniqueName: \"kubernetes.io/projected/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5-kube-api-access-4m7ds\") pod \"kas-connection-checker-685895c7c9-txjqv\" (UID: \"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5\") " pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:03.713144 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.713117    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vm2\" (UniqueName: \"kubernetes.io/projected/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911-kube-api-access-f8vm2\") pod \"kas-connection-checker-685895c7c9-7dsfq\" (UID: \"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911\") " pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:03.726280 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.726242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:03.734748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.734269    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:45:03.734748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.734453    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\""
Nov 03 17:45:03.745713 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.745853 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbdf\" (UniqueName: \"kubernetes.io/projected/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-kube-api-access-8cbdf\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.745853 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-tmp\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.745853 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.746031 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.746031 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-snapshots\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.746031 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-stats-auth\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.746031 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9524c\" (UniqueName: \"kubernetes.io/projected/3ae4fdcf-846d-464a-a6a5-28713517c9aa-kube-api-access-9524c\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.746031 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.745995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr87j\" (UniqueName: \"kubernetes.io/projected/9ef120cf-f11d-435c-96c4-96d73f511afa-kube-api-access-tr87j\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.746031 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae4fdcf-846d-464a-a6a5-28713517c9aa-tmp-dir\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.746328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-default-certificate\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.746328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.746328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-service-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.746328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khf9l\" (UniqueName: \"kubernetes.io/projected/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-kube-api-access-khf9l\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.746328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxwt\" (UniqueName: \"kubernetes.io/projected/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-kube-api-access-wzxwt\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.746328 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae4fdcf-846d-464a-a6a5-28713517c9aa-config-volume\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.746839 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746807    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-service-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.746920 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746883    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-snapshots\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.746973 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/2fd2bd81-c082-46e4-87c2-226f1acf6c07-kube-api-access-5th4k\") pod \"network-check-source-6f78d59f99-nwhnp\" (UID: \"2fd2bd81-c082-46e4-87c2-226f1acf6c07\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:03.747032 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.746994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45jf\" (UniqueName: \"kubernetes.io/projected/16fc5076-624a-47e0-9430-9550882714d8-kube-api-access-p45jf\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.747085 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.747085 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.747207 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.747207 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.747207 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.747207 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.747394 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747333    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-tmp\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.747455 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.747455 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef120cf-f11d-435c-96c4-96d73f511afa-serving-cert\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.747553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.747553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.747553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.747553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.747537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-cert\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.748118 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748096    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.748213 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-config\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.748269 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-serving-cert\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.748269 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.748453 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748421    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.748685 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.748911 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.748886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-config\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.749216 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.749171    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.749310 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.749257    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:03.749310 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.749272    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c466dc58-kmk8g: secret "image-registry-tls" not found
Nov 03 17:45:03.750580 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.749320    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls podName:22f8456e-5749-4025-9161-4609e71ef8e8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.249304272 +0000 UTC m=+34.378300844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls") pod "image-registry-69c466dc58-kmk8g" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8") : secret "image-registry-tls" not found
Nov 03 17:45:03.750580 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.749689    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.751849 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.751480    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 17:45:03.751849 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.751553    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls podName:16fc5076-624a-47e0-9430-9550882714d8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.25152487 +0000 UTC m=+34.380521447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rx64c" (UID: "16fc5076-624a-47e0-9430-9550882714d8") : secret "samples-operator-tls" not found
Nov 03 17:45:03.752523 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.752504    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef120cf-f11d-435c-96c4-96d73f511afa-serving-cert\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.752686 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.752633    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.752928 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.752890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.753020 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.752973    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.753595 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.753553    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-serving-cert\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.798735 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.798427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:03.814468 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.812609    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr87j\" (UniqueName: \"kubernetes.io/projected/9ef120cf-f11d-435c-96c4-96d73f511afa-kube-api-access-tr87j\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.817991 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.817958    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbdf\" (UniqueName: \"kubernetes.io/projected/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-kube-api-access-8cbdf\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:03.819955 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.819930    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.821899 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.821446    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/2fd2bd81-c082-46e4-87c2-226f1acf6c07-kube-api-access-5th4k\") pod \"network-check-source-6f78d59f99-nwhnp\" (UID: \"2fd2bd81-c082-46e4-87c2-226f1acf6c07\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:03.824327 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.824192    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45jf\" (UniqueName: \"kubernetes.io/projected/16fc5076-624a-47e0-9430-9550882714d8-kube-api-access-p45jf\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:03.827064 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.827041    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:03.849318 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.849318 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.849490 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-cert\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.849490 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.849553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-stats-auth\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.849553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9524c\" (UniqueName: \"kubernetes.io/projected/3ae4fdcf-846d-464a-a6a5-28713517c9aa-kube-api-access-9524c\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.849608 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae4fdcf-846d-464a-a6a5-28713517c9aa-tmp-dir\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.849608 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-default-certificate\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.849665 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khf9l\" (UniqueName: \"kubernetes.io/projected/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-kube-api-access-khf9l\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.849700 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxwt\" (UniqueName: \"kubernetes.io/projected/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-kube-api-access-wzxwt\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.849700 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.849687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae4fdcf-846d-464a-a6a5-28713517c9aa-config-volume\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.850363 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.850215    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 17:45:03.850363 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.850275    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.35026067 +0000 UTC m=+34.479257240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : secret "router-metrics-certs-default" not found
Nov 03 17:45:03.850363 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.850337    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.350330276 +0000 UTC m=+34.479326849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : configmap references non-existent config key: service-ca.crt
Nov 03 17:45:03.850597 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.850372    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae4fdcf-846d-464a-a6a5-28713517c9aa-config-volume\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.850650 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.850606    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:45:03.850695 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:03.850651    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls podName:3ae4fdcf-846d-464a-a6a5-28713517c9aa nodeName:}" failed. No retries permitted until 2026-11-03 17:45:04.350637126 +0000 UTC m=+34.479633706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls") pod "dns-default-lrmc7" (UID: "3ae4fdcf-846d-464a-a6a5-28713517c9aa") : secret "dns-default-metrics-tls" not found
Nov 03 17:45:03.850858 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.850829    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae4fdcf-846d-464a-a6a5-28713517c9aa-tmp-dir\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.852942 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.852916    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-default-certificate\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.853746 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.853691    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-stats-auth\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.854016 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.854000    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-cert\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.855811 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.855795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:03.911149 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.910296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:03.917453 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.916342    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khf9l\" (UniqueName: \"kubernetes.io/projected/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-kube-api-access-khf9l\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:03.917453 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.916709    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxwt\" (UniqueName: \"kubernetes.io/projected/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-kube-api-access-wzxwt\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:03.918004 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.917942    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9524c\" (UniqueName: \"kubernetes.io/projected/3ae4fdcf-846d-464a-a6a5-28713517c9aa-kube-api-access-9524c\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:03.974592 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:03.974555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:04.003632 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.003606    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:04.098091 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.098014    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e00a5d_ac2d_4dc5_90cf_106cf9a4bf52.slice/crio-18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a WatchSource:0}: Error finding container 18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a: Status 404 returned error can't find the container with id 18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a
Nov 03 17:45:04.100057 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.099690    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"]
Nov 03 17:45:04.101514 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.101490    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9392d9f2_6a5c_4b6a_b048_a20227a79836.slice/crio-f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65 WatchSource:0}: Error finding container f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65: Status 404 returned error can't find the container with id f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65
Nov 03 17:45:04.103267 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.103108    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65f6r"]
Nov 03 17:45:04.105188 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.104095    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"]
Nov 03 17:45:04.105188 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.104470    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4803ee_0fee_44d8_a5d9_7931656d6e7c.slice/crio-e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac WatchSource:0}: Error finding container e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac: Status 404 returned error can't find the container with id e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac
Nov 03 17:45:04.112610 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.112590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:04.144467 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.144439    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4e187c_93a8_4dc1_bd70_4c4dfc3b2fb5.slice/crio-623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e WatchSource:0}: Error finding container 623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e: Status 404 returned error can't find the container with id 623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e
Nov 03 17:45:04.146303 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.146281    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txjqv"]
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.153551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.153641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.153679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.153810    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.153864    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls podName:69cbe72d-a018-4a89-8951-a25e50d6d33b nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.153847803 +0000 UTC m=+35.282844373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8ff8h" (UID: "69cbe72d-a018-4a89-8951-a25e50d6d33b") : secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.153945    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.153955    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6589fbf699-mdv92: secret "image-registry-tls" not found
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.153987    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls podName:18d852d5-b4c4-4f8f-b2da-bc84b039025e nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.153976599 +0000 UTC m=+35.282973170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls") pod "image-registry-6589fbf699-mdv92" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e") : secret "image-registry-tls" not found
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.154041    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 17:45:04.154088 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.154071    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert podName:08a559aa-cebd-40a6-9393-3feb287029ee nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.154062981 +0000 UTC m=+35.283059539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkbq7" (UID: "08a559aa-cebd-40a6-9393-3feb287029ee") : secret "networking-console-plugin-cert" not found
Nov 03 17:45:04.195015 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.194993    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"]
Nov 03 17:45:04.199965 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.199856    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae3db47_36e8_4957_9c10_fd361815db36.slice/crio-d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83 WatchSource:0}: Error finding container d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83: Status 404 returned error can't find the container with id d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83
Nov 03 17:45:04.228873 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.228788    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd2bd81_c082_46e4_87c2_226f1acf6c07.slice/crio-942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d WatchSource:0}: Error finding container 942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d: Status 404 returned error can't find the container with id 942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d
Nov 03 17:45:04.228873 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.228854    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"]
Nov 03 17:45:04.254340 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.254298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:04.254534 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254458    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:45:04.254621 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.254546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:04.254621 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254592    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs podName:d5703035-ada3-4cfb-b77b-4a8e9bcbe063 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:36.254569033 +0000 UTC m=+66.383565603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs") pod "network-metrics-daemon-8b682" (UID: "d5703035-ada3-4cfb-b77b-4a8e9bcbe063") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:45:04.254621 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254611    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:45:04.254885 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254662    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret podName:9807c6c6-ef8b-4769-bf2e-97b5c5a23604 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:36.254648812 +0000 UTC m=+66.383645382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret") pod "global-pull-secret-syncer-zc2rm" (UID: "9807c6c6-ef8b-4769-bf2e-97b5c5a23604") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:45:04.254885 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.254703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:04.254885 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.254767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:04.255123 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254909    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:04.255123 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254920    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c466dc58-kmk8g: secret "image-registry-tls" not found
Nov 03 17:45:04.255123 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.254957    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls podName:22f8456e-5749-4025-9161-4609e71ef8e8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.25494703 +0000 UTC m=+35.383943603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls") pod "image-registry-69c466dc58-kmk8g" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8") : secret "image-registry-tls" not found
Nov 03 17:45:04.255123 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.255071    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 17:45:04.255313 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.255130    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls podName:16fc5076-624a-47e0-9430-9550882714d8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.255115881 +0000 UTC m=+35.384112453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rx64c" (UID: "16fc5076-624a-47e0-9430-9550882714d8") : secret "samples-operator-tls" not found
Nov 03 17:45:04.294434 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.294402    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef120cf_f11d_435c_96c4_96d73f511afa.slice/crio-b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8 WatchSource:0}: Error finding container b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8: Status 404 returned error can't find the container with id b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8
Nov 03 17:45:04.296359 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.296335    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6nngz"]
Nov 03 17:45:04.355671 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.355641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:04.355671 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.355673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:04.355871 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.355736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:04.355871 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.355824    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:45:04.355871 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.355823    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.355805641 +0000 UTC m=+35.484802218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : configmap references non-existent config key: service-ca.crt
Nov 03 17:45:04.355962 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.355873    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 17:45:04.355962 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.355908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:04.355962 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.355925    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.355912956 +0000 UTC m=+35.484909540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : secret "router-metrics-certs-default" not found
Nov 03 17:45:04.355962 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:04.355951    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls podName:3ae4fdcf-846d-464a-a6a5-28713517c9aa nodeName:}" failed. No retries permitted until 2026-11-03 17:45:05.355944507 +0000 UTC m=+35.484941067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls") pod "dns-default-lrmc7" (UID: "3ae4fdcf-846d-464a-a6a5-28713517c9aa") : secret "dns-default-metrics-tls" not found
Nov 03 17:45:04.359514 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.359490    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:04.368518 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.368491    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904ebe0f_c56d_4bd5_aae8_53dd560c8f5c.slice/crio-b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d WatchSource:0}: Error finding container b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d: Status 404 returned error can't find the container with id b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d
Nov 03 17:45:04.371049 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.371024    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nkxg8"]
Nov 03 17:45:04.397170 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.397147    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8110d47e_1f9c_480d_a1b6_d4a2b2b1e911.slice/crio-bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95 WatchSource:0}: Error finding container bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95: Status 404 returned error can't find the container with id bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95
Nov 03 17:45:04.399425 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.399405    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7dsfq"]
Nov 03 17:45:04.468701 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.468668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:04.468868 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.468747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:04.468868 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.468746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:04.517920 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.517892    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f389b73_c992_49ac_ba2d_cf69b91f2b1e.slice/crio-da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b WatchSource:0}: Error finding container da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b: Status 404 returned error can't find the container with id da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b
Nov 03 17:45:04.520645 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.520626    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"]
Nov 03 17:45:04.525165 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.525142    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:45:04.525295 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.525274    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\""
Nov 03 17:45:04.525361 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.525348    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:45:04.525428 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.525346    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\""
Nov 03 17:45:04.525474 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.525427    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\""
Nov 03 17:45:04.533516 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.533498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:04.661572 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.661407    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dsfq" event={"ID":"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911","Type":"ContainerStarted","Data":"22f1a73ba0f9faceb94a17eeedf19c17bed1bc83474c5b404ec27ff0c9b04431"}
Nov 03 17:45:04.661752 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.661603    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7dsfq" event={"ID":"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911","Type":"ContainerStarted","Data":"bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95"}
Nov 03 17:45:04.662477 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.662448    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds" event={"ID":"bae3db47-36e8-4957-9c10-fd361815db36","Type":"ContainerStarted","Data":"d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83"}
Nov 03 17:45:04.663747 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.663698    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-txjqv" event={"ID":"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5","Type":"ContainerStarted","Data":"5ecaf448c4c148b588fcb1ae6ef51d3800aace5fd0ffccbabe9dfc8ed820a465"}
Nov 03 17:45:04.663747 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.663745    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-txjqv" event={"ID":"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5","Type":"ContainerStarted","Data":"623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e"}
Nov 03 17:45:04.664825 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.664806    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm" event={"ID":"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52","Type":"ContainerStarted","Data":"18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a"}
Nov 03 17:45:04.665878 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.665858    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" event={"ID":"1f389b73-c992-49ac-ba2d-cf69b91f2b1e","Type":"ContainerStarted","Data":"da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b"}
Nov 03 17:45:04.666869 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.666850    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nkxg8" event={"ID":"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c","Type":"ContainerStarted","Data":"b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d"}
Nov 03 17:45:04.667816 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.667794    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6nngz" event={"ID":"9ef120cf-f11d-435c-96c4-96d73f511afa","Type":"ContainerStarted","Data":"b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8"}
Nov 03 17:45:04.668765 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.668745    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp" event={"ID":"2fd2bd81-c082-46e4-87c2-226f1acf6c07","Type":"ContainerStarted","Data":"942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d"}
Nov 03 17:45:04.670212 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.670194    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65f6r" event={"ID":"9392d9f2-6a5c-4b6a-b048-a20227a79836","Type":"ContainerStarted","Data":"faedfdc2066b721214128969962b7fef20f606c6fb7364d0c5136ad114d66012"}
Nov 03 17:45:04.670299 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.670227    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65f6r" event={"ID":"9392d9f2-6a5c-4b6a-b048-a20227a79836","Type":"ContainerStarted","Data":"f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65"}
Nov 03 17:45:04.671349 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.671329    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c" event={"ID":"4e4803ee-0fee-44d8-a5d9-7931656d6e7c","Type":"ContainerStarted","Data":"e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac"}
Nov 03 17:45:04.911387 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:04.911360    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d645b31_5684_42a1_b305_59fe985149c4.slice/crio-3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c WatchSource:0}: Error finding container 3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c: Status 404 returned error can't find the container with id 3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c
Nov 03 17:45:04.913783 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:04.913762    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jrt48"]
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.164292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.164346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.164432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164585    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164600    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6589fbf699-mdv92: secret "image-registry-tls" not found
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls podName:18d852d5-b4c4-4f8f-b2da-bc84b039025e nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.164638159 +0000 UTC m=+37.293634722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls") pod "image-registry-6589fbf699-mdv92" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e") : secret "image-registry-tls" not found
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164729    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164770    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert podName:08a559aa-cebd-40a6-9393-3feb287029ee nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.164757955 +0000 UTC m=+37.293754526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkbq7" (UID: "08a559aa-cebd-40a6-9393-3feb287029ee") : secret "networking-console-plugin-cert" not found
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164831    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:05.164936 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.164867    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls podName:69cbe72d-a018-4a89-8951-a25e50d6d33b nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.164856485 +0000 UTC m=+37.293853057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8ff8h" (UID: "69cbe72d-a018-4a89-8951-a25e50d6d33b") : secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:05.228496 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.228429    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7dsfq" podStartSLOduration=363.228407583 podStartE2EDuration="6m3.228407583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:05.105759365 +0000 UTC m=+35.234755944" watchObservedRunningTime="2026-11-03 17:45:05.228407583 +0000 UTC m=+35.357404163"
Nov 03 17:45:05.265859 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.265826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:05.266026 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.265988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:05.266026 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.266003    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:05.266026 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.266025    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c466dc58-kmk8g: secret "image-registry-tls" not found
Nov 03 17:45:05.266176 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.266092    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls podName:22f8456e-5749-4025-9161-4609e71ef8e8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.266070153 +0000 UTC m=+37.395066723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls") pod "image-registry-69c466dc58-kmk8g" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8") : secret "image-registry-tls" not found
Nov 03 17:45:05.266176 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.266092    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 17:45:05.266176 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.266148    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls podName:16fc5076-624a-47e0-9430-9550882714d8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.266136656 +0000 UTC m=+37.395133227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rx64c" (UID: "16fc5076-624a-47e0-9430-9550882714d8") : secret "samples-operator-tls" not found
Nov 03 17:45:05.366775 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.366742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:05.367030 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.366903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:05.367030 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.366933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:05.367224 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.367077    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 17:45:05.367224 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.367135    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.367119222 +0000 UTC m=+37.496115797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : secret "router-metrics-certs-default" not found
Nov 03 17:45:05.367224 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.367190    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:45:05.367224 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.367219    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls podName:3ae4fdcf-846d-464a-a6a5-28713517c9aa nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.367211156 +0000 UTC m=+37.496207712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls") pod "dns-default-lrmc7" (UID: "3ae4fdcf-846d-464a-a6a5-28713517c9aa") : secret "dns-default-metrics-tls" not found
Nov 03 17:45:05.367428 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:05.367276    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.36727013 +0000 UTC m=+37.496266699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : configmap references non-existent config key: service-ca.crt
Nov 03 17:45:05.680746 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.680531    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jrt48" event={"ID":"8d645b31-5684-42a1-b305-59fe985149c4","Type":"ContainerStarted","Data":"3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c"}
Nov 03 17:45:05.718422 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:05.718316    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-txjqv" podStartSLOduration=363.718297716 podStartE2EDuration="6m3.718297716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:05.591455234 +0000 UTC m=+35.720451814" watchObservedRunningTime="2026-11-03 17:45:05.718297716 +0000 UTC m=+35.847294307"
Nov 03 17:45:07.186345 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.186274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:07.186345 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.186341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:07.186987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.186554    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 17:45:07.186987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.186624    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert podName:08a559aa-cebd-40a6-9393-3feb287029ee nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.186604653 +0000 UTC m=+41.315601228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkbq7" (UID: "08a559aa-cebd-40a6-9393-3feb287029ee") : secret "networking-console-plugin-cert" not found
Nov 03 17:45:07.186987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.186699    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:07.186987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.186821    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls podName:69cbe72d-a018-4a89-8951-a25e50d6d33b nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.186803213 +0000 UTC m=+41.315799771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8ff8h" (UID: "69cbe72d-a018-4a89-8951-a25e50d6d33b") : secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:07.187345 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.187323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:07.187541 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.187517    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:07.187541 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.187535    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6589fbf699-mdv92: secret "image-registry-tls" not found
Nov 03 17:45:07.187710 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.187575    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls podName:18d852d5-b4c4-4f8f-b2da-bc84b039025e nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.187563234 +0000 UTC m=+41.316559805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls") pod "image-registry-6589fbf699-mdv92" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e") : secret "image-registry-tls" not found
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.288815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.288869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.289048    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.289062    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c466dc58-kmk8g: secret "image-registry-tls" not found
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.289132    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls podName:22f8456e-5749-4025-9161-4609e71ef8e8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.289104474 +0000 UTC m=+41.418101045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls") pod "image-registry-69c466dc58-kmk8g" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8") : secret "image-registry-tls" not found
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.289200    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 17:45:07.289286 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.289231    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls podName:16fc5076-624a-47e0-9430-9550882714d8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.289221221 +0000 UTC m=+41.418217792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rx64c" (UID: "16fc5076-624a-47e0-9430-9550882714d8") : secret "samples-operator-tls" not found
Nov 03 17:45:07.390072 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.389996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:07.390072 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.390053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:07.390255 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.390116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:07.390351 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.390332    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:45:07.390406 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.390401    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls podName:3ae4fdcf-846d-464a-a6a5-28713517c9aa nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.390383337 +0000 UTC m=+41.519379913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls") pod "dns-default-lrmc7" (UID: "3ae4fdcf-846d-464a-a6a5-28713517c9aa") : secret "dns-default-metrics-tls" not found
Nov 03 17:45:07.390492 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.390478    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.390468326 +0000 UTC m=+41.519464886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : configmap references non-existent config key: service-ca.crt
Nov 03 17:45:07.390564 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.390541    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 17:45:07.390610 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:07.390573    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:11.390563111 +0000 UTC m=+41.519559682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : secret "router-metrics-certs-default" not found
Nov 03 17:45:07.697635 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.697597    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c" event={"ID":"4e4803ee-0fee-44d8-a5d9-7931656d6e7c","Type":"ContainerStarted","Data":"031d891fe70f56cffd6cde46ac80b0b3a3a410b462cee091ca0c2be4bc313937"}
Nov 03 17:45:07.821137 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:07.820393    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-65f6r" podStartSLOduration=365.820371408 podStartE2EDuration="6m5.820371408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:05.839981118 +0000 UTC m=+35.968977698" watchObservedRunningTime="2026-11-03 17:45:07.820371408 +0000 UTC m=+37.949367989"
Nov 03 17:45:10.708862 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.708832    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/0.log"
Nov 03 17:45:10.709553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.708872    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" containerID="7dd6772a20aba60bda54da5da56a6982f6355b0da313fb672593abc4c1c766b8" exitCode=255
Nov 03 17:45:10.709553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.708955    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" event={"ID":"1f389b73-c992-49ac-ba2d-cf69b91f2b1e","Type":"ContainerDied","Data":"7dd6772a20aba60bda54da5da56a6982f6355b0da313fb672593abc4c1c766b8"}
Nov 03 17:45:10.709553 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.709207    2581 scope.go:122] "RemoveContainer" containerID="7dd6772a20aba60bda54da5da56a6982f6355b0da313fb672593abc4c1c766b8"
Nov 03 17:45:10.710979 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.710951    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nkxg8" event={"ID":"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c","Type":"ContainerStarted","Data":"54ad88df6d5d2c82f981863a1efd905c49801a0907cd4441a170d63ba1c16add"}
Nov 03 17:45:10.712951 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.712931    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6nngz" event={"ID":"9ef120cf-f11d-435c-96c4-96d73f511afa","Type":"ContainerStarted","Data":"0156fd83b62140dc89698ce0d83ebf758abdf90acd5d1720c815f83bc2eeaaba"}
Nov 03 17:45:10.714891 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.714864    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp" event={"ID":"2fd2bd81-c082-46e4-87c2-226f1acf6c07","Type":"ContainerStarted","Data":"1a4ffbadd24c04478d098d724b77cc09a01602fd897e3d23c13453db9b4641b7"}
Nov 03 17:45:10.717003 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.716978    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds" event={"ID":"bae3db47-36e8-4957-9c10-fd361815db36","Type":"ContainerStarted","Data":"4d35fe6754dc2bb67af5ba9e41e5d73fa29dd392c62db10c6cf50cc89a312e7e"}
Nov 03 17:45:10.719147 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.719125    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm" event={"ID":"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52","Type":"ContainerStarted","Data":"8df8ab03b27a871a8bd2dc934fae9f42b35dd273020a91039cf8f2b3909fb98a"}
Nov 03 17:45:10.834545 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.834380    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c" podStartSLOduration=330.751014391 podStartE2EDuration="5m33.834367882s" totalImagesPullingTime="3.083353491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:07.821089251 +0000 UTC m=+37.950085834" watchObservedRunningTime="2026-11-03 17:45:10.834367882 +0000 UTC m=+40.963364460"
Nov 03 17:45:10.958001 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:10.957940    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds" podStartSLOduration=356.128586484 podStartE2EDuration="6m1.957925283s" totalImagesPullingTime="5.829338799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:10.956487279 +0000 UTC m=+41.085483876" watchObservedRunningTime="2026-11-03 17:45:10.957925283 +0000 UTC m=+41.086921861"
Nov 03 17:45:11.079123 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.079075    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp" podStartSLOduration=321.793017707 podStartE2EDuration="5m28.079062202s" totalImagesPullingTime="6.286044495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:11.078862115 +0000 UTC m=+41.207858711" watchObservedRunningTime="2026-11-03 17:45:11.079062202 +0000 UTC m=+41.208058781"
Nov 03 17:45:11.201424 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.201311    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nkxg8" podStartSLOduration=2.067475083 podStartE2EDuration="8.20129641s" totalImagesPullingTime="6.133821327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:11.200176227 +0000 UTC m=+41.329172826" watchObservedRunningTime="2026-11-03 17:45:11.20129641 +0000 UTC m=+41.330292991"
Nov 03 17:45:11.233525 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.233490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:11.233706 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.233539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:11.233706 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233630    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 17:45:11.233706 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.233675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:11.233706 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233694    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert podName:08a559aa-cebd-40a6-9393-3feb287029ee nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.233675483 +0000 UTC m=+49.362672046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkbq7" (UID: "08a559aa-cebd-40a6-9393-3feb287029ee") : secret "networking-console-plugin-cert" not found
Nov 03 17:45:11.233706 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233694    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:11.233993 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233761    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:11.233993 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233766    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls podName:69cbe72d-a018-4a89-8951-a25e50d6d33b nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.233752508 +0000 UTC m=+49.362749069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8ff8h" (UID: "69cbe72d-a018-4a89-8951-a25e50d6d33b") : secret "cluster-monitoring-operator-tls" not found
Nov 03 17:45:11.233993 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233774    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6589fbf699-mdv92: secret "image-registry-tls" not found
Nov 03 17:45:11.233993 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.233812    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls podName:18d852d5-b4c4-4f8f-b2da-bc84b039025e nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.233796838 +0000 UTC m=+49.362793395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls") pod "image-registry-6589fbf699-mdv92" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e") : secret "image-registry-tls" not found
Nov 03 17:45:11.327202 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.327148    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm" podStartSLOduration=350.394016192 podStartE2EDuration="5m56.327132153s" totalImagesPullingTime="5.933115961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:11.326155097 +0000 UTC m=+41.455151687" watchObservedRunningTime="2026-11-03 17:45:11.327132153 +0000 UTC m=+41.456128742"
Nov 03 17:45:11.334740 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.334697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:11.334881 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.334779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:11.334881 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.334835    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 17:45:11.334987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.334903    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls podName:16fc5076-624a-47e0-9430-9550882714d8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.334885195 +0000 UTC m=+49.463881775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rx64c" (UID: "16fc5076-624a-47e0-9430-9550882714d8") : secret "samples-operator-tls" not found
Nov 03 17:45:11.334987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.334909    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 17:45:11.334987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.334924    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c466dc58-kmk8g: secret "image-registry-tls" not found
Nov 03 17:45:11.334987 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.334952    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls podName:22f8456e-5749-4025-9161-4609e71ef8e8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.334944261 +0000 UTC m=+49.463940841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls") pod "image-registry-69c466dc58-kmk8g" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8") : secret "image-registry-tls" not found
Nov 03 17:45:11.383009 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.382980    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hprg4"]
Nov 03 17:45:11.397576 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.397554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.435910 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.435883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:11.435910 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.435910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:11.436112 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.435941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:11.436112 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.436056    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 17:45:11.436112 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.436068    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.436045633 +0000 UTC m=+49.565042204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : configmap references non-existent config key: service-ca.crt
Nov 03 17:45:11.436112 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.436100    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs podName:2947ed62-ecb7-4ccd-935b-b1870cffb3e5 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.436088551 +0000 UTC m=+49.565085111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs") pod "router-default-7b45cc6f6f-ph9d6" (UID: "2947ed62-ecb7-4ccd-935b-b1870cffb3e5") : secret "router-metrics-certs-default" not found
Nov 03 17:45:11.436112 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.436056    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:45:11.436285 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.436150    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls podName:3ae4fdcf-846d-464a-a6a5-28713517c9aa nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.436137738 +0000 UTC m=+49.565134317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls") pod "dns-default-lrmc7" (UID: "3ae4fdcf-846d-464a-a6a5-28713517c9aa") : secret "dns-default-metrics-tls" not found
Nov 03 17:45:11.443765 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.443739    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hprg4"]
Nov 03 17:45:11.450542 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.450501    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-6nngz" podStartSLOduration=352.713965308 podStartE2EDuration="5m58.45048775s" totalImagesPullingTime="5.736522442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:11.449137837 +0000 UTC m=+41.578134437" watchObservedRunningTime="2026-11-03 17:45:11.45048775 +0000 UTC m=+41.579484326"
Nov 03 17:45:11.454982 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.454931    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:45:11.454982 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.454975    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvbgk\""
Nov 03 17:45:11.536991 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.536963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.537143 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.537004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad0fe61-4d34-46c0-9b9f-263eb91dc103-data-volume\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.537143 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.537040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aad0fe61-4d34-46c0-9b9f-263eb91dc103-crio-socket\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.537231 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.537212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.537271 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.537240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblft\" (UniqueName: \"kubernetes.io/projected/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-api-access-mblft\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.549325 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.549296    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:45:11.632823 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.632791    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4pbtv-w76t9"]
Nov 03 17:45:11.638401 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638517 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad0fe61-4d34-46c0-9b9f-263eb91dc103-data-volume\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638517 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aad0fe61-4d34-46c0-9b9f-263eb91dc103-crio-socket\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638517 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.638501    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:45:11.638667 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.638550    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:12.138537618 +0000 UTC m=+42.267534179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : configmap "kube-rbac-proxy" not found
Nov 03 17:45:11.638748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638748 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblft\" (UniqueName: \"kubernetes.io/projected/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-api-access-mblft\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638846 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638742    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aad0fe61-4d34-46c0-9b9f-263eb91dc103-crio-socket\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638846 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.638791    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad0fe61-4d34-46c0-9b9f-263eb91dc103-data-volume\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.638846 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.638831    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:11.638928 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.638882    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:12.138871918 +0000 UTC m=+42.267868479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:11.664532 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.664508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.708088 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.708014    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4pbtv-w76t9"]
Nov 03 17:45:11.719064 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.719038    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblft\" (UniqueName: \"kubernetes.io/projected/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-api-access-mblft\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:11.727135 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.727114    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:45:11.727838 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.727818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/0.log"
Nov 03 17:45:11.727955 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.727862    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" containerID="24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842" exitCode=255
Nov 03 17:45:11.728017 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.727947    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" event={"ID":"1f389b73-c992-49ac-ba2d-cf69b91f2b1e","Type":"ContainerDied","Data":"24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842"}
Nov 03 17:45:11.728071 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.728027    2581 scope.go:122] "RemoveContainer" containerID="7dd6772a20aba60bda54da5da56a6982f6355b0da313fb672593abc4c1c766b8"
Nov 03 17:45:11.728202 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.728184    2581 scope.go:122] "RemoveContainer" containerID="24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842"
Nov 03 17:45:11.728484 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:11.728459    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-w2mkr_openshift-console-operator(1f389b73-c992-49ac-ba2d-cf69b91f2b1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" podUID="1f389b73-c992-49ac-ba2d-cf69b91f2b1e"
Nov 03 17:45:11.728713 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.728698    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jd7fn\""
Nov 03 17:45:11.731465 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.731274    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jrt48" event={"ID":"8d645b31-5684-42a1-b305-59fe985149c4","Type":"ContainerStarted","Data":"8b7ddd68c7e1bcb084edda66dc78ed73e0e7c787df4040ad2649be6101df0af8"}
Nov 03 17:45:11.731934 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.731912    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:11.840579 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.840540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.840771 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.840673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.840914 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.840881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.841390 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.841372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.854242 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.854217    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"]
Nov 03 17:45:11.880736 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.880702    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"]
Nov 03 17:45:11.880873 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.880836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:11.896867 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.896844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:11.914147 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.914118    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"]
Nov 03 17:45:11.924447 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.924424    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"]
Nov 03 17:45:11.942386 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.942358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.942524 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.942418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.942524 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.942513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.942634 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.942601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.942919 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.942893    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.943299 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.943246    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.945539 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.945513    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:11.946464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.946439    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:11.946464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.946440    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 17:45:11.946590 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:11.946472    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rfn9h\""
Nov 03 17:45:12.015362 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.015278    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:12.043248 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.043220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hxz\" (UniqueName: \"kubernetes.io/projected/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00-kube-api-access-p8hxz\") pod \"migrator-66df5d496b-k6qjm\" (UID: \"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:12.043439 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.043422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kv8n\" (UniqueName: \"kubernetes.io/projected/63acd3e4-e973-4124-afcf-39f855f0e4aa-kube-api-access-7kv8n\") pod \"migrator-66df5d496b-cnjjw\" (UID: \"63acd3e4-e973-4124-afcf-39f855f0e4aa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:12.052879 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.052828    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jrt48" podStartSLOduration=34.973116522 podStartE2EDuration="41.052814621s" totalImagesPullingTime="6.079698099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:12.051788121 +0000 UTC m=+42.180784702" watchObservedRunningTime="2026-11-03 17:45:12.052814621 +0000 UTC m=+42.181811200"
Nov 03 17:45:12.147794 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.147763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:12.147983 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.147853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kv8n\" (UniqueName: \"kubernetes.io/projected/63acd3e4-e973-4124-afcf-39f855f0e4aa-kube-api-access-7kv8n\") pod \"migrator-66df5d496b-cnjjw\" (UID: \"63acd3e4-e973-4124-afcf-39f855f0e4aa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:12.147983 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.147926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:12.147983 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:12.147937    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:12.147983 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.147948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hxz\" (UniqueName: \"kubernetes.io/projected/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00-kube-api-access-p8hxz\") pod \"migrator-66df5d496b-k6qjm\" (UID: \"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:12.148214 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:12.148013    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:13.147987897 +0000 UTC m=+43.276984467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:12.148214 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:12.148050    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:45:12.148214 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:12.148107    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:13.14809281 +0000 UTC m=+43.277089382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : configmap "kube-rbac-proxy" not found
Nov 03 17:45:12.212103 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.212068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kv8n\" (UniqueName: \"kubernetes.io/projected/63acd3e4-e973-4124-afcf-39f855f0e4aa-kube-api-access-7kv8n\") pod \"migrator-66df5d496b-cnjjw\" (UID: \"63acd3e4-e973-4124-afcf-39f855f0e4aa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:12.212237 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.212068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hxz\" (UniqueName: \"kubernetes.io/projected/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00-kube-api-access-p8hxz\") pod \"migrator-66df5d496b-k6qjm\" (UID: \"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:12.221953 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.221932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:12.288892 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.288829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:12.503396 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.503358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:12.623655 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:12.623627    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63acd3e4_e973_4124_afcf_39f855f0e4aa.slice/crio-ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d WatchSource:0}: Error finding container ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d: Status 404 returned error can't find the container with id ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d
Nov 03 17:45:12.627601 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.627580    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"]
Nov 03 17:45:12.686910 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:12.686883    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef95306f_a39f_419e_84d7_6496c754701b.slice/crio-fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9 WatchSource:0}: Error finding container fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9: Status 404 returned error can't find the container with id fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9
Nov 03 17:45:12.690403 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.690383    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4pbtv-w76t9"]
Nov 03 17:45:12.735358 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.735332    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4pbtv-w76t9" event={"ID":"ef95306f-a39f-419e-84d7-6496c754701b","Type":"ContainerStarted","Data":"fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9"}
Nov 03 17:45:12.736808 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.736790    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:45:12.737130 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.737108    2581 scope.go:122] "RemoveContainer" containerID="24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842"
Nov 03 17:45:12.737312 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:12.737290    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-w2mkr_openshift-console-operator(1f389b73-c992-49ac-ba2d-cf69b91f2b1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" podUID="1f389b73-c992-49ac-ba2d-cf69b91f2b1e"
Nov 03 17:45:12.737974 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.737951    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw" event={"ID":"63acd3e4-e973-4124-afcf-39f855f0e4aa","Type":"ContainerStarted","Data":"ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d"}
Nov 03 17:45:12.896708 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:12.896678    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4eaf38_4ab2_4a04_91e5_bb2e9b184c00.slice/crio-91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c WatchSource:0}: Error finding container 91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c: Status 404 returned error can't find the container with id 91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c
Nov 03 17:45:12.900133 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:12.900112    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"]
Nov 03 17:45:13.007160 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.007125    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m5xh8"]
Nov 03 17:45:13.024938 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.024913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.056578 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.056549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-key\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.056690 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.056596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.056784 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.056768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbv9j\" (UniqueName: \"kubernetes.io/projected/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-kube-api-access-jbv9j\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.066076 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.066052    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m5xh8"]
Nov 03 17:45:13.080494 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.080463    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 17:45:13.080601 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.080495    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 17:45:13.080601 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.080532    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 17:45:13.080698 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.080607    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 17:45:13.080698 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.080617    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bx8p5\""
Nov 03 17:45:13.157402 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.157338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbv9j\" (UniqueName: \"kubernetes.io/projected/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-kube-api-access-jbv9j\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.157402 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.157375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.157405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-key\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.157438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.157479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:13.157504    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:13.157550    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:13.157574    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:15.157555112 +0000 UTC m=+45.286551680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:13.157618 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:13.157593    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:15.157581495 +0000 UTC m=+45.286578053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : configmap "kube-rbac-proxy" not found
Nov 03 17:45:13.158155 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.158138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.160573 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.160552    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-key\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.222871 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.222840    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbv9j\" (UniqueName: \"kubernetes.io/projected/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-kube-api-access-jbv9j\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.343714 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.343678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:13.741544 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:13.741291    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b03107f_d2b0_4a5d_b404_9dd5bccb7dc7.slice/crio-28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07 WatchSource:0}: Error finding container 28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07: Status 404 returned error can't find the container with id 28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07
Nov 03 17:45:13.743313 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.743285    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-m5xh8"]
Nov 03 17:45:13.743757 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.743673    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm" event={"ID":"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00","Type":"ContainerStarted","Data":"91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c"}
Nov 03 17:45:13.746295 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.746267    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4pbtv-w76t9" event={"ID":"ef95306f-a39f-419e-84d7-6496c754701b","Type":"ContainerStarted","Data":"3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd"}
Nov 03 17:45:13.884956 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:13.884893    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4pbtv-w76t9" podStartSLOduration=2.884873592 podStartE2EDuration="2.884873592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:13.88334235 +0000 UTC m=+44.012338930" watchObservedRunningTime="2026-11-03 17:45:13.884873592 +0000 UTC m=+44.013870176"
Nov 03 17:45:14.113585 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.113552    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:14.113777 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.113630    2581 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:14.114158 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.114135    2581 scope.go:122] "RemoveContainer" containerID="24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842"
Nov 03 17:45:14.114745 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:14.114409    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-w2mkr_openshift-console-operator(1f389b73-c992-49ac-ba2d-cf69b91f2b1e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" podUID="1f389b73-c992-49ac-ba2d-cf69b91f2b1e"
Nov 03 17:45:14.751647 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.751424    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw" event={"ID":"63acd3e4-e973-4124-afcf-39f855f0e4aa","Type":"ContainerStarted","Data":"dda3cafba4a2dd601bb5ee50ebe142e0269a4c450a8685a4fb53d53f0f70ec81"}
Nov 03 17:45:14.752935 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.752909    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8" event={"ID":"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7","Type":"ContainerStarted","Data":"80e2e0046ed556d7935dc305b2d29dba18593ef5fb6e9c162b82db7ef5aa7f1c"}
Nov 03 17:45:14.753017 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.752954    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8" event={"ID":"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7","Type":"ContainerStarted","Data":"28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07"}
Nov 03 17:45:14.872688 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:14.872636    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw" podStartSLOduration=1.866878069 podStartE2EDuration="3.872619386s" totalImagesPullingTime="2.005741317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:14.872611696 +0000 UTC m=+45.001608276" watchObservedRunningTime="2026-11-03 17:45:14.872619386 +0000 UTC m=+45.001615967"
Nov 03 17:45:15.001374 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.001249    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8" podStartSLOduration=3.001234023 podStartE2EDuration="3.001234023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:15.00081226 +0000 UTC m=+45.129808866" watchObservedRunningTime="2026-11-03 17:45:15.001234023 +0000 UTC m=+45.130230603"
Nov 03 17:45:15.177226 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.177189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:15.177376 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.177363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:15.177548 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:15.177520    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:15.177672 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:15.177602    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls podName:aad0fe61-4d34-46c0-9b9f-263eb91dc103 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:19.177582161 +0000 UTC m=+49.306578735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hprg4" (UID: "aad0fe61-4d34-46c0-9b9f-263eb91dc103") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:45:15.177993 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.177794    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:15.565589 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.565557    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7mnmh_b9984e6b-2191-4627-96ac-9d0b53762a46/dns-node-resolver/0.log"
Nov 03 17:45:15.757951 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.757917    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm" event={"ID":"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00","Type":"ContainerStarted","Data":"e68a5d921cfe50d3f3abc4b409c95f05c72e0aea7b5597cb1b40baafcecc1a0a"}
Nov 03 17:45:15.885339 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:15.885247    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm" podStartSLOduration=3.020503575 podStartE2EDuration="4.885231784s" totalImagesPullingTime="1.864728209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:15.885142564 +0000 UTC m=+46.014139143" watchObservedRunningTime="2026-11-03 17:45:15.885231784 +0000 UTC m=+46.014228363"
Nov 03 17:45:16.565937 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:16.565910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8z24k_94ef9fc0-8b2b-440a-bdd1-5684a5bc573d/node-ca/0.log"
Nov 03 17:45:17.365908 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:17.365879    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nkxg8_904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/serve-healthcheck-canary/0.log"
Nov 03 17:45:17.568489 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:17.568453    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-cnjjw_63acd3e4-e973-4124-afcf-39f855f0e4aa/migrator/0.log"
Nov 03 17:45:17.766556 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:17.766476    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-k6qjm_fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00/migrator/0.log"
Nov 03 17:45:17.966862 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:17.966837    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-7mpds_bae3db47-36e8-4957-9c10-fd361815db36/kube-storage-version-migrator-operator/0.log"
Nov 03 17:45:19.214895 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.214852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:19.218429 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.218405    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:19.250136 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.250112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:19.316254 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.316222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:19.316342 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.316268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:19.316387 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.316351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:19.316930 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:19.316518    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 17:45:19.316930 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:19.316592    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert podName:08a559aa-cebd-40a6-9393-3feb287029ee nodeName:}" failed. No retries permitted until 2026-11-03 17:45:35.316575728 +0000 UTC m=+65.445572285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kkbq7" (UID: "08a559aa-cebd-40a6-9393-3feb287029ee") : secret "networking-console-plugin-cert" not found
Nov 03 17:45:19.319516 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.319492    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:19.320317 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.320290    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:19.349988 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.349966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:19.369816 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.369788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:19.417479 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.417442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:19.417624 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.417506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:19.417787 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:19.417767    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 17:45:19.417864 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:19.417845    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls podName:16fc5076-624a-47e0-9430-9550882714d8 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:35.417828056 +0000 UTC m=+65.546824628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rx64c" (UID: "16fc5076-624a-47e0-9430-9550882714d8") : secret "samples-operator-tls" not found
Nov 03 17:45:19.421125 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.421087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:19.437185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.437011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:19.518121 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.518093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:19.518256 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.518197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:19.518256 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.518215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:19.518256 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:19.518250    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:45:19.518396 ip-10-0-141-108 kubenswrapper[2581]: E1103 17:45:19.518314    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls podName:3ae4fdcf-846d-464a-a6a5-28713517c9aa nodeName:}" failed. No retries permitted until 2026-11-03 17:45:35.518297412 +0000 UTC m=+65.647293984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls") pod "dns-default-lrmc7" (UID: "3ae4fdcf-846d-464a-a6a5-28713517c9aa") : secret "dns-default-metrics-tls" not found
Nov 03 17:45:19.519033 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.519012    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:19.520429 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.520409    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:19.529427 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.529409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:19.618706 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:19.618676    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad0fe61_4d34_46c0_9b9f_263eb91dc103.slice/crio-eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b WatchSource:0}: Error finding container eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b: Status 404 returned error can't find the container with id eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b
Nov 03 17:45:19.623005 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.622982    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hprg4"]
Nov 03 17:45:19.730223 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:19.730194    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d852d5_b4c4_4f8f_b2da_bc84b039025e.slice/crio-1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858 WatchSource:0}: Error finding container 1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858: Status 404 returned error can't find the container with id 1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858
Nov 03 17:45:19.732489 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.732468    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92"]
Nov 03 17:45:19.772783 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.772741    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" event={"ID":"18d852d5-b4c4-4f8f-b2da-bc84b039025e","Type":"ContainerStarted","Data":"1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858"}
Nov 03 17:45:19.772988 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:19.772963    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cbe72d_a018_4a89_8951_a25e50d6d33b.slice/crio-c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663 WatchSource:0}: Error finding container c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663: Status 404 returned error can't find the container with id c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663
Nov 03 17:45:19.774285 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.774267    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hprg4" event={"ID":"aad0fe61-4d34-46c0-9b9f-263eb91dc103","Type":"ContainerStarted","Data":"17fa73dbfec23391182bd78354964be1ebff041faf7ae1f55020f59fd96cc6d4"}
Nov 03 17:45:19.774344 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.774316    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hprg4" event={"ID":"aad0fe61-4d34-46c0-9b9f-263eb91dc103","Type":"ContainerStarted","Data":"eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b"}
Nov 03 17:45:19.777077 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.777058    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"]
Nov 03 17:45:19.829921 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:19.829888    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f8456e_5749_4025_9161_4609e71ef8e8.slice/crio-58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d WatchSource:0}: Error finding container 58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d: Status 404 returned error can't find the container with id 58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d
Nov 03 17:45:19.832040 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.832016    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c466dc58-kmk8g"]
Nov 03 17:45:19.905100 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:19.905068    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2947ed62_ecb7_4ccd_935b_b1870cffb3e5.slice/crio-f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a WatchSource:0}: Error finding container f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a: Status 404 returned error can't find the container with id f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a
Nov 03 17:45:19.908263 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:19.908243    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b45cc6f6f-ph9d6"]
Nov 03 17:45:20.779836 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.779798    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" event={"ID":"22f8456e-5749-4025-9161-4609e71ef8e8","Type":"ContainerStarted","Data":"3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17"}
Nov 03 17:45:20.780249 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.779858    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" event={"ID":"22f8456e-5749-4025-9161-4609e71ef8e8","Type":"ContainerStarted","Data":"58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d"}
Nov 03 17:45:20.780249 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.780060    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:20.781813 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.781777    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6" event={"ID":"2947ed62-ecb7-4ccd-935b-b1870cffb3e5","Type":"ContainerStarted","Data":"7b019d1db7997e6ce3246f492bde04707fee6243d3986ba976e9d1d0fa2edf4e"}
Nov 03 17:45:20.781939 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.781825    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6" event={"ID":"2947ed62-ecb7-4ccd-935b-b1870cffb3e5","Type":"ContainerStarted","Data":"f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a"}
Nov 03 17:45:20.784324 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.784298    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" event={"ID":"18d852d5-b4c4-4f8f-b2da-bc84b039025e","Type":"ContainerStarted","Data":"9ac86bccc6e897224c7b089768857317d7fa72c929475687f4ee97d5dce6f597"}
Nov 03 17:45:20.784400 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.784384    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:20.785854 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.785831    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h" event={"ID":"69cbe72d-a018-4a89-8951-a25e50d6d33b","Type":"ContainerStarted","Data":"c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663"}
Nov 03 17:45:20.787896 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.787873    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hprg4" event={"ID":"aad0fe61-4d34-46c0-9b9f-263eb91dc103","Type":"ContainerStarted","Data":"0f76d09f483d6cb9e58c55baa688b6798c330875bbeec5f35021ad4600ecc725"}
Nov 03 17:45:20.910907 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:20.910848    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" podStartSLOduration=40.910830728 podStartE2EDuration="40.910830728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:20.909108944 +0000 UTC m=+51.038105523" watchObservedRunningTime="2026-11-03 17:45:20.910830728 +0000 UTC m=+51.039827308"
Nov 03 17:45:21.037530 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:21.037434    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" podStartSLOduration=344.037419211 podStartE2EDuration="5m44.037419211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:21.03668994 +0000 UTC m=+51.165686665" watchObservedRunningTime="2026-11-03 17:45:21.037419211 +0000 UTC m=+51.166415784"
Nov 03 17:45:21.165008 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:21.164950    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6" podStartSLOduration=367.164932306 podStartE2EDuration="6m7.164932306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:21.164128554 +0000 UTC m=+51.293125161" watchObservedRunningTime="2026-11-03 17:45:21.164932306 +0000 UTC m=+51.293928884"
Nov 03 17:45:21.530128 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:21.530093    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:21.533125 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:21.533098    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:21.794012 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:21.793931    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:21.795079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:21.795031    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:22.798460 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:22.798423    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h" event={"ID":"69cbe72d-a018-4a89-8951-a25e50d6d33b","Type":"ContainerStarted","Data":"f8179459200c5321e2e0c346de49446cd3286504ca2c59085d0d36ed62a2be92"}
Nov 03 17:45:22.800771 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:22.800745    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hprg4" event={"ID":"aad0fe61-4d34-46c0-9b9f-263eb91dc103","Type":"ContainerStarted","Data":"1c516a2ae1ae3c6f6c80edd403b99dc67d56aff98b9d12bf925d9710b56727de"}
Nov 03 17:45:23.061479 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:23.061433    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hprg4" podStartSLOduration=9.242391965 podStartE2EDuration="12.061417748s" totalImagesPullingTime="2.819025783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:45:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:23.060467102 +0000 UTC m=+53.189463704" watchObservedRunningTime="2026-11-03 17:45:23.061417748 +0000 UTC m=+53.190414327"
Nov 03 17:45:23.061693 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:23.061673    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h" podStartSLOduration=343.981490314 podStartE2EDuration="5m46.061666527s" totalImagesPullingTime="2.080176213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:22.930828211 +0000 UTC m=+53.059824810" watchObservedRunningTime="2026-11-03 17:45:23.061666527 +0000 UTC m=+53.190663106"
Nov 03 17:45:23.469026 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:23.468984    2581 scope.go:122] "RemoveContainer" containerID="24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842"
Nov 03 17:45:23.806018 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:23.805950    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:45:23.806367 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:23.806089    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" event={"ID":"1f389b73-c992-49ac-ba2d-cf69b91f2b1e","Type":"ContainerStarted","Data":"56e85deff71bff7dd4194f29fae9fdd0aeeb566d55ac7fb55d0fd9039e02877f"}
Nov 03 17:45:23.930955 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:23.930903    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr" podStartSLOduration=369.417087072 podStartE2EDuration="6m14.930886464s" totalImagesPullingTime="5.513799392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:23.930133846 +0000 UTC m=+54.059130444" watchObservedRunningTime="2026-11-03 17:45:23.930886464 +0000 UTC m=+54.059883043"
Nov 03 17:45:26.653711 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:26.653684    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:33.806620 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:33.806585    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:33.810249 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:33.810227    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:34.251683 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.251652    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vfvbk"]
Nov 03 17:45:34.257034 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.257007    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"]
Nov 03 17:45:34.257183 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.257155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.260335 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.260312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:34.260444 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.260420    2581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92"]
Nov 03 17:45:34.266397 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.266370    2581 patch_prober.go:29] interesting pod/image-registry-6589fbf699-mdv92 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:34.266513 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.266420    2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:34.320807 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.320782    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 17:45:34.321275 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.321250    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 17:45:34.321698 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.321682    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xlmkc\""
Nov 03 17:45:34.324205 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.324189    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6rhnc\""
Nov 03 17:45:34.338228 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.338206    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd4cffc4-zvgw7"]
Nov 03 17:45:34.341463 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.341442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.344298 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.344278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfpt\" (UniqueName: \"kubernetes.io/projected/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-kube-api-access-bxfpt\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.344375 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.344334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/36506792-bfe4-4a9f-b27c-c2096cb67ebe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v9k7l\" (UID: \"36506792-bfe4-4a9f-b27c-c2096cb67ebe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:34.344375 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.344352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-tmp\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.358958 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.358937    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"]
Nov 03 17:45:34.362246 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.362220    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vfvbk"]
Nov 03 17:45:34.414918 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.411776    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd4cffc4-zvgw7"]
Nov 03 17:45:34.424829 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.424796    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 17:45:34.445302 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357c52c-00a5-4bb2-bf8e-315d3760e129-ca-trust-extracted\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445302 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-bound-sa-token\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-certificates\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-image-registry-private-configuration\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445464 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-tls\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445566 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/36506792-bfe4-4a9f-b27c-c2096cb67ebe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v9k7l\" (UID: \"36506792-bfe4-4a9f-b27c-c2096cb67ebe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:34.445566 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-tmp\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.445643 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmrv\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-kube-api-access-2fmrv\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445690 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-installation-pull-secrets\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445755 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-trusted-ca\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.445872 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfpt\" (UniqueName: \"kubernetes.io/projected/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-kube-api-access-bxfpt\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.445995 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.445976    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-tmp\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.447922 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.447895    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/36506792-bfe4-4a9f-b27c-c2096cb67ebe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v9k7l\" (UID: \"36506792-bfe4-4a9f-b27c-c2096cb67ebe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:34.519518 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.519451    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfpt\" (UniqueName: \"kubernetes.io/projected/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-kube-api-access-bxfpt\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.546447 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-installation-pull-secrets\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546564 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-trusted-ca\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546564 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357c52c-00a5-4bb2-bf8e-315d3760e129-ca-trust-extracted\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546564 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-bound-sa-token\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546564 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-certificates\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546564 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-image-registry-private-configuration\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546798 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-tls\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.546798 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmrv\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-kube-api-access-2fmrv\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.547003 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.546978    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357c52c-00a5-4bb2-bf8e-315d3760e129-ca-trust-extracted\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.547415 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.547385    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-certificates\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.547512 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.547479    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-trusted-ca\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.548857 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.548829    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-installation-pull-secrets\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.548970 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.548943    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-image-registry-private-configuration\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.549316 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.549294    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-tls\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.572460 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.572440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:34.577550 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.577532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:34.610787 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.610702    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-bound-sa-token\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.611143 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.611121    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmrv\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-kube-api-access-2fmrv\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.655990 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.655711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:34.695999 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.695962    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-574b5dd6dd-f4459"]
Nov 03 17:45:34.699818 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.699788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.755781 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.755751    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 17:45:34.755942 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.755824    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 17:45:34.755942 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.755855    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5hg79\""
Nov 03 17:45:34.755942 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.755859    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 17:45:34.755942 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.755900    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 17:45:34.756084 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.756000    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 17:45:34.756461 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.756441    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574b5dd6dd-f4459"]
Nov 03 17:45:34.849868 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.849834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.850267 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.849889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.850267 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.849977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.850267 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.850087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.850267 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.850113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.850267 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.850155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.850539 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.850278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.951699 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.951668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.951699 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.951697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.951904 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.951844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.951904 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.951876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.951995 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.951962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.952061 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.951992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.952061 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.952032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.952381 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.952347    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.952488 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.952399    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.952654 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.952636    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.952692 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.952649    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.954643 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.954618    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.954709 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.954686    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:34.992349 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.992325    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"]
Nov 03 17:45:34.993021 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:34.992999    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4548c83e_c9e2_45fe_8ec0_3e450486cfd9.slice/crio-5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172 WatchSource:0}: Error finding container 5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172: Status 404 returned error can't find the container with id 5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172
Nov 03 17:45:34.995861 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:34.995796    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vfvbk"]
Nov 03 17:45:35.015883 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.015863    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:35.043559 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:35.043531    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4357c52c_00a5_4bb2_bf8e_315d3760e129.slice/crio-e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b WatchSource:0}: Error finding container e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b: Status 404 returned error can't find the container with id e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b
Nov 03 17:45:35.047468 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.047450    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd4cffc4-zvgw7"]
Nov 03 17:45:35.315781 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.315755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:35.355138 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.355102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:35.357567 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.357542    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:35.456415 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.456377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:35.458917 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.458886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:35.557123 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.557089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:35.559559 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.559537    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:35.637004 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.636933    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7lgsk\""
Nov 03 17:45:35.642994 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.642954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:35.685192 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:35.685166    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc23c25a_4638_4c06_8804_3b1fe5135d56.slice/crio-3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95 WatchSource:0}: Error finding container 3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95: Status 404 returned error can't find the container with id 3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95
Nov 03 17:45:35.688185 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.688161    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574b5dd6dd-f4459"]
Nov 03 17:45:35.692150 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.691984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:35.851406 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.851383    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\""
Nov 03 17:45:35.851865 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.851588    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7" event={"ID":"4357c52c-00a5-4bb2-bf8e-315d3760e129","Type":"ContainerStarted","Data":"db82feb359deef35b6be83c177c3feeec9211b50768d2ecc7c7d77cee2312b5c"}
Nov 03 17:45:35.851865 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.851633    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7" event={"ID":"4357c52c-00a5-4bb2-bf8e-315d3760e129","Type":"ContainerStarted","Data":"e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b"}
Nov 03 17:45:35.851865 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.851745    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:35.852855 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.852831    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574b5dd6dd-f4459" event={"ID":"fc23c25a-4638-4c06-8804-3b1fe5135d56","Type":"ContainerStarted","Data":"3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95"}
Nov 03 17:45:35.853939 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.853914    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vfvbk" event={"ID":"4548c83e-c9e2-45fe-8ec0-3e450486cfd9","Type":"ContainerStarted","Data":"5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172"}
Nov 03 17:45:35.854962 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.854939    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l" event={"ID":"36506792-bfe4-4a9f-b27c-c2096cb67ebe","Type":"ContainerStarted","Data":"e445f6815d1c366d2ee1fd82bad71272edaac6496102dc6f47fc11934510b77a"}
Nov 03 17:45:35.857129 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:35.857114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:36.013187 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.013127    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7" podStartSLOduration=2.01310996 podStartE2EDuration="2.01310996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:35.98010722 +0000 UTC m=+66.109103799" watchObservedRunningTime="2026-11-03 17:45:36.01310996 +0000 UTC m=+66.142106535"
Nov 03 17:45:36.014079 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.014059    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"]
Nov 03 17:45:36.025919 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:36.025887    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a559aa_cebd_40a6_9393_3feb287029ee.slice/crio-c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b WatchSource:0}: Error finding container c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b: Status 404 returned error can't find the container with id c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b
Nov 03 17:45:36.069789 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.069760    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"]
Nov 03 17:45:36.226995 ip-10-0-141-108 kubenswrapper[2581]: W1103 17:45:36.226959    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae4fdcf_846d_464a_a6a5_28713517c9aa.slice/crio-3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95 WatchSource:0}: Error finding container 3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95: Status 404 returned error can't find the container with id 3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95
Nov 03 17:45:36.233357 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.233329    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lrmc7"]
Nov 03 17:45:36.264093 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.264062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:36.264262 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.264120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:36.319032 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.318990    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:45:36.323964 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.323929    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:45:36.328689 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.328659    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:36.330692 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.330666    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:36.390412 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.390382    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\""
Nov 03 17:45:36.396470 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.396430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:36.682261 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.682007    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\""
Nov 03 17:45:36.687772 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.687590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:36.783982 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.783868    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8b682"]
Nov 03 17:45:36.861750 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.861652    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c" event={"ID":"16fc5076-624a-47e0-9430-9550882714d8","Type":"ContainerStarted","Data":"7937eb9a256b8eecceedb6a37a2e57b50d9269ee4b70b6475a390ade72203c81"}
Nov 03 17:45:36.863835 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.863808    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l" event={"ID":"36506792-bfe4-4a9f-b27c-c2096cb67ebe","Type":"ContainerStarted","Data":"726a003bc174ab84ac2236821f874dda13edc9bbbf95ef68cb3c0c0feaa8c397"}
Nov 03 17:45:36.864669 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.864647    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:36.866900 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.866851    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b682" event={"ID":"d5703035-ada3-4cfb-b77b-4a8e9bcbe063","Type":"ContainerStarted","Data":"b09440f78484f7bfc73778c34b13dcd3535e8374c6c8a200fb9c897f4dfcf298"}
Nov 03 17:45:36.868594 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.868539    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrmc7" event={"ID":"3ae4fdcf-846d-464a-a6a5-28713517c9aa","Type":"ContainerStarted","Data":"3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95"}
Nov 03 17:45:36.870850 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.870827    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:36.872500 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.872479    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7" event={"ID":"08a559aa-cebd-40a6-9393-3feb287029ee","Type":"ContainerStarted","Data":"c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b"}
Nov 03 17:45:36.987097 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:36.987030    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l" podStartSLOduration=1.369508607 podStartE2EDuration="2.987010896s" totalImagesPullingTime="1.617502289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:36.985455226 +0000 UTC m=+67.114451807" watchObservedRunningTime="2026-11-03 17:45:36.987010896 +0000 UTC m=+67.116007479"
Nov 03 17:45:37.070692 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:37.070663    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zc2rm"]
Nov 03 17:45:37.883481 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:37.883441    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zc2rm" event={"ID":"9807c6c6-ef8b-4769-bf2e-97b5c5a23604","Type":"ContainerStarted","Data":"a83dcd85a67a063d980e6dc6f7f08905992bb16e94094130e3e0af6f378225fa"}
Nov 03 17:45:39.442576 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.442551    2581 patch_prober.go:29] interesting pod/image-registry-69c466dc58-kmk8g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:39.442970 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.442592    2581 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:39.902404 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.902311    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7" event={"ID":"08a559aa-cebd-40a6-9393-3feb287029ee","Type":"ContainerStarted","Data":"a65f3302f2a9f87cf62c0b5ffd9335347038df82be683f9f155bc568b9c675d3"}
Nov 03 17:45:39.906754 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.906703    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c" event={"ID":"16fc5076-624a-47e0-9430-9550882714d8","Type":"ContainerStarted","Data":"63b210a94db07e80af05df83fa54dbd767f6b677ab234d3bd9ceaaf220e57552"}
Nov 03 17:45:39.906880 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.906771    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c" event={"ID":"16fc5076-624a-47e0-9430-9550882714d8","Type":"ContainerStarted","Data":"0c4dafa5a781cb5b80b648840672c7dfffc7499b924a48571046bc44b4614662"}
Nov 03 17:45:39.910287 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.910260    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b682" event={"ID":"d5703035-ada3-4cfb-b77b-4a8e9bcbe063","Type":"ContainerStarted","Data":"218546fe45d320a40086becdc524dc8e26cef0df852d2b74c70c06900ec625e1"}
Nov 03 17:45:39.910406 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:39.910312    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b682" event={"ID":"d5703035-ada3-4cfb-b77b-4a8e9bcbe063","Type":"ContainerStarted","Data":"99a46f706c559b3b02ef60bf3ee4f9f1cd61aef9173b069607ae4b44edf229d2"}
Nov 03 17:45:40.032732 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:40.032664    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7" podStartSLOduration=347.548562823 podStartE2EDuration="5m51.032644325s" totalImagesPullingTime="3.484081502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:40.024026777 +0000 UTC m=+70.153023396" watchObservedRunningTime="2026-11-03 17:45:40.032644325 +0000 UTC m=+70.161640907"
Nov 03 17:45:40.275001 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:40.274950    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c" podStartSLOduration=386.074902668 podStartE2EDuration="6m29.274934164s" totalImagesPullingTime="3.200031496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:40.274048143 +0000 UTC m=+70.403044746" watchObservedRunningTime="2026-11-03 17:45:40.274934164 +0000 UTC m=+70.403930743"
Nov 03 17:45:40.275737 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:40.275687    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8b682" podStartSLOduration=67.052308787 podStartE2EDuration="1m9.275673946s" totalImagesPullingTime="2.223365159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:40.148684425 +0000 UTC m=+70.277681007" watchObservedRunningTime="2026-11-03 17:45:40.275673946 +0000 UTC m=+70.404670526"
Nov 03 17:45:40.920074 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:40.919219    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrmc7" event={"ID":"3ae4fdcf-846d-464a-a6a5-28713517c9aa","Type":"ContainerStarted","Data":"0280f58337b9c0a6b0b5356a9283751db1ff8492d560a48846af3dbb10961d69"}
Nov 03 17:45:40.920074 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:40.919290    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lrmc7" event={"ID":"3ae4fdcf-846d-464a-a6a5-28713517c9aa","Type":"ContainerStarted","Data":"34e201521feee2f6e76c9cc4d69d7e7c063e777ae43789f9c0df00f0913f4b0a"}
Nov 03 17:45:40.920074 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:40.919514    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:41.044571 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:41.044222    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lrmc7" podStartSLOduration=34.361420153 podStartE2EDuration="38.044201704s" totalImagesPullingTime="3.682781551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.042988234 +0000 UTC m=+71.171984828" watchObservedRunningTime="2026-11-03 17:45:41.044201704 +0000 UTC m=+71.173198286"
Nov 03 17:45:41.799324 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:41.799287    2581 patch_prober.go:29] interesting pod/image-registry-69c466dc58-kmk8g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:41.799624 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:41.799344    2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:41.932265 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:41.932216    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574b5dd6dd-f4459" event={"ID":"fc23c25a-4638-4c06-8804-3b1fe5135d56","Type":"ContainerStarted","Data":"685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e"}
Nov 03 17:45:42.062588 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:42.062477    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-574b5dd6dd-f4459" podStartSLOduration=2.728641066 podStartE2EDuration="8.06245842s" totalImagesPullingTime="5.333817354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:42.061575064 +0000 UTC m=+72.190571663" watchObservedRunningTime="2026-11-03 17:45:42.06245842 +0000 UTC m=+72.191455031"
Nov 03 17:45:42.740827 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:42.740792    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:44.267052 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.266701    2581 patch_prober.go:29] interesting pod/image-registry-6589fbf699-mdv92 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:44.267052 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.266798    2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:44.515102 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.515066    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6gk44"]
Nov 03 17:45:44.534042 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.533976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646361 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646538 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646538 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-textfile\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646538 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdnc\" (UniqueName: \"kubernetes.io/projected/33b55ac5-4039-4da9-81f2-b64fadc22435-kube-api-access-dtdnc\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-sys\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-wtmp\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-metrics-client-ca\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646703 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-tls\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.646922 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.646731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-root\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:44.654613 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.654588    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:45:44.654831 ip-10-0-141-108 kubenswrapper[2581]: I1103 17:45:44.654815    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:45:44.692252 ip-10-0-141-108 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 17:45:44.707209 ip-10-0-141-108 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 17:45:44.707436 ip-10-0-141-108 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 17:45:44.707632 ip-10-0-141-108 systemd[1]: kubelet.service: Consumed 4.972s CPU time.
Nov 03 17:45:44.727371 ip-10-0-141-108 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:45:44.823811 ip-10-0-141-108 kubenswrapper[6167]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:45:44.823811 ip-10-0-141-108 kubenswrapper[6167]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:45:44.823811 ip-10-0-141-108 kubenswrapper[6167]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:45:44.823811 ip-10-0-141-108 kubenswrapper[6167]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:45:44.825994 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.825911    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.825994 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.825994    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.825994 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.825998    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826002    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826004    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826008    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826011    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826014    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826016    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826019    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826021    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826024    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826027    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826031    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826035    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826039    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826044    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826048    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826052    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826057    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826060    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826070    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.826099 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826074    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826077    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826079    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826083    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826085    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826088    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826091    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826094    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826099    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826103    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826107    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826111    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826114    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826117    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826120    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826122    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826125    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826128    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826131    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.826615 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826134    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826139    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826143    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826146    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826149    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826152    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826156    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826159    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826162    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826164    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826167    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826170    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826172    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826175    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826177    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826180    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826183    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826185    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826188    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.827110 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826191    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826195    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826198    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826200    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826203    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826206    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826208    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826211    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826213    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826216    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826218    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826221    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826224    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826226    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826229    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826232    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826235    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826238    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826240    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826243    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.827650 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826245    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826248    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826251    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826253    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826256    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826259    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826261    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826264    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826267    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826269    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826272    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826274    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826280    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826283    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826286    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826289    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826292    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826294    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826297    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826300    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.828197 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826302    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826305    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826307    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826310    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826312    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.826315    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827139    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827147    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827150    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827156    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827160    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827163    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827166    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827169    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827172    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827175    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827177    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827180    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827184    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.828738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827188    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827191    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827193    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827196    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827199    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827202    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827205    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827208    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827211    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827213    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827216    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827221    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827225    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827229    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827233    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827237    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827241    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827245    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827249    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827252    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.829257 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827254    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827257    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827260    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827263    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827266    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827268    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827271    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827274    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827277    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827280    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827282    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827285    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827288    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827290    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827294    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827296    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827299    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827302    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827304    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827308    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.829845 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827313    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827317    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827322    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827325    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827328    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827330    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827333    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827336    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827339    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827341    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827344    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827347    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827350    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827352    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827355    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827358    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827361    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827364    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827367    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.830367 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827369    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827372    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827374    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827377    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827382    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827386    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827390    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827394    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827398    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827401    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827404    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827408    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827411    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827413    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827416    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827419    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827421    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827424    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827426    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.830888 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827429    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827432    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827434    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827437    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827439    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827443    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827445    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827448    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827451    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827453    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827455    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827458    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827462    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827464    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827467    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827967    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827976    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827979    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827981    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827984    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.831378 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827987    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827989    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827992    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827995    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.827998    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828001    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828003    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828006    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828009    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828012    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828015    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828017    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828020    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828023    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828025    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828028    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828031    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828033    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828036    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.832277 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828039    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828041    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828044    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828046    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828049    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828052    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828055    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828057    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828060    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828064    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828067    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828069    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828072    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828074    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828077    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828079    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828082    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828085    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828088    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828090    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.832786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828093    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828095    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828098    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828100    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828103    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828105    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828108    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828111    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828113    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828116    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828119    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828121    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828124    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828126    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828129    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828132    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828135    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828139    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828142    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.833285 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828145    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828147    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828151    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828154    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828158    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828160    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828163    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828165    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828168    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828170    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828173    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828176    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828178    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828181    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828184    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828186    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828189    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828191    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828194    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828197    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.833801 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828199    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828202    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828205    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828208    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828210    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828213    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828215    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828218    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828220    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828223    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828225    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828228    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828231    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828234    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828238    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828243    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828246    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828249    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828251    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.834306 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828254    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828256    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828260    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.828263    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828353    6167 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828360    6167 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828366    6167 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828371    6167 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828375    6167 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828378    6167 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828385    6167 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828389    6167 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828393    6167 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828396    6167 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828399    6167 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828403    6167 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828406    6167 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828409    6167 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828412    6167 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828415    6167 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828418    6167 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828421    6167 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828426    6167 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828429    6167 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:45:44.834794 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828432    6167 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828436    6167 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828439    6167 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828444    6167 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828448    6167 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828452    6167 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828455    6167 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828459    6167 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828462    6167 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828465    6167 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828468    6167 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828471    6167 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828475    6167 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828478    6167 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828481    6167 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828484    6167 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828487    6167 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828491    6167 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828495    6167 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828498    6167 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828501    6167 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828505    6167 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828508    6167 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828512    6167 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:45:44.835415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828515    6167 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828518    6167 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828521    6167 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828525    6167 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828528    6167 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828531    6167 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828534    6167 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828537    6167 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828540    6167 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828543    6167 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828548    6167 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828551    6167 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828556    6167 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828559    6167 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828564    6167 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828567    6167 flags.go:64] FLAG: --help="false"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828570    6167 flags.go:64] FLAG: --hostname-override="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828573    6167 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828576    6167 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828580    6167 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828583    6167 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828587    6167 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828590    6167 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:45:44.836089 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828592    6167 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828595    6167 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828598    6167 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828601    6167 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828605    6167 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828608    6167 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828611    6167 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828614    6167 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828617    6167 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828620    6167 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828623    6167 flags.go:64] FLAG: --lock-file=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828625    6167 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828628    6167 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828631    6167 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828637    6167 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828640    6167 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828643    6167 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828646    6167 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828649    6167 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828652    6167 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828656    6167 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828660    6167 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828665    6167 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828670    6167 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828674    6167 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:45:44.837046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828677    6167 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828681    6167 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828684    6167 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828687    6167 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828690    6167 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828693    6167 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828696    6167 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828702    6167 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828705    6167 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828709    6167 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828712    6167 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828729    6167 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828733    6167 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828736    6167 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828739    6167 flags.go:64] FLAG: --port="10250"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828742    6167 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828745    6167 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0eeb938a8f227e418"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828749    6167 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828752    6167 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828755    6167 flags.go:64] FLAG: --register-node="true"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828758    6167 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828762    6167 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828765    6167 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828768    6167 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828771    6167 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828775    6167 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:45:44.838003 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828778    6167 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828781    6167 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828787    6167 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828790    6167 flags.go:64] FLAG: --runonce="false"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828794    6167 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828797    6167 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828800    6167 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828803    6167 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828806    6167 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828810    6167 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828813    6167 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828817    6167 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828820    6167 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828822    6167 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828826    6167 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828828    6167 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828832    6167 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828835    6167 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828838    6167 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828843    6167 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828846    6167 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828849    6167 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828853    6167 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828856    6167 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828859    6167 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:45:44.838762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828862    6167 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828865    6167 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828868    6167 flags.go:64] FLAG: --v="2"
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828872    6167 flags.go:64] FLAG: --version="false"
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828877    6167 flags.go:64] FLAG: --vmodule=""
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828880    6167 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.828884    6167 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829016    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829019    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829024    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829028    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829033    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829036    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829039    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829041    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829044    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829046    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829049    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829052    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829055    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.839426 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829058    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829062    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829065    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829068    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829071    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829074    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829077    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829079    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829083    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829085    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829088    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829091    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829094    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829096    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829099    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829101    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829104    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829107    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829109    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829112    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.840055 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829115    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829119    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829121    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829126    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829128    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829131    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829134    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829136    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829139    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829142    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829145    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829147    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829150    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829153    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829157    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829160    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829163    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829166    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829168    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.840733 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829171    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829174    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829177    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829179    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829182    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829184    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829187    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829190    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829192    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829195    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829197    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829200    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829202    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829205    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829210    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829212    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829216    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829219    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829222    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829225    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.841405 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829227    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829230    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829233    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829235    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829238    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829241    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829243    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829246    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829249    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829251    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829254    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829256    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829259    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829261    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829264    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829267    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829270    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829273    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829275    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829278    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.842114 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829281    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829283    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829286    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829288    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829291    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829294    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829297    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829300    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829304    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829307    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829310    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829313    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829315    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829318    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.829343    6167 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:45:44.842632 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829376    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829381    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829384    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829387    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829390    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829392    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829395    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829397    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829400    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829403    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829405    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829408    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829410    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829413    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829415    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829418    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829421    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829423    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829426    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829428    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829431    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.843224 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829434    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829437    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829441    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829443    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829446    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829453    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829457    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829461    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829464    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829467    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829470    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829473    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829476    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829479    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829482    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829485    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829487    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829489    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829492    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.843913 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829495    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829497    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829500    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829502    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829505    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829507    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829510    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829512    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829515    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829518    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829520    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829523    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829525    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829528    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829530    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829534    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829537    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829540    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829543    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829546    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.844834 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829548    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829551    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829554    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829557    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829560    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829562    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829565    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829567    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829570    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829573    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829575    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829578    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829580    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829583    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829586    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829589    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829592    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829595    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829597    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.845812 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829600    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829602    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829605    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829607    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829610    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829612    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829615    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829618    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829622    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829625    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829628    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829632    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829634    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829637    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829640    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829643    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829645    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829648    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829650    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.846620 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829655    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829658    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829661    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829664    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829667    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829669    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829672    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.829675    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.833862    6167 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.833878    6167 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.833897    6167 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.833902    6167 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.833981    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.833987    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.833991    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.833994    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.833996    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.833999    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834003    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.847495 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834007    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834011    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834013    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834016    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834019    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834022    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834025    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834027    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834030    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834040    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834043    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834046    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834049    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834052    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834054    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834057    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834059    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834062    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834065    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.848360 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834068    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834070    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834073    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834076    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834078    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834081    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834083    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834086    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834090    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834094    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834097    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834099    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834102    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834105    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834108    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834110    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834113    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834116    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834118    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.849190 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834121    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834124    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834126    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834129    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834137    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834140    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834143    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834146    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834148    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834151    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834154    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834156    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834159    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834162    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834164    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834167    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834169    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834172    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834174    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834177    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.850100 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834179    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834183    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834185    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834188    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834191    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834193    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834196    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834208    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834211    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834214    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834216    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834219    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834221    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834224    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834226    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834229    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834232    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834240    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834243    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834245    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.851366 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834248    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834251    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834253    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834256    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834258    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834261    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834263    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834266    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834268    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834271    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834273    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834276    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834279    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834281    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834284    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834286    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834289    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834291    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834294    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834297    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.852264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834299    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.834326    6167 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834517    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834521    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834525    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834528    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834532    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834535    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834538    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834541    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834549    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834552    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834555    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834557    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834560    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834562    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.853154 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834565    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834568    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834570    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834573    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834575    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834578    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834580    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834584    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834586    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834589    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834591    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834594    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834596    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834599    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834601    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834604    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834607    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834609    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834611    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.853867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834614    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834617    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834619    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834622    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834625    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834627    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834630    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834632    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834640    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834643    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834646    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834648    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834651    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834653    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834656    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834658    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834661    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834664    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834666    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.854619 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834669    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834671    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834674    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834677    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834679    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834682    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834684    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834687    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834689    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834692    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834700    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834703    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834705    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834708    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834710    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834713    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834740    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834744    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834746    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834749    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.855157 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834752    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834754    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834764    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834767    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834771    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834775    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834778    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834780    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834784    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834788    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834790    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834793    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834796    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834798    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834801    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834804    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834807    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834810    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834813    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.855661 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834815    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834818    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834820    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834823    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834825    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834828    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834830    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834833    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834835    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834838    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834841    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834844    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834846    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834849    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834851    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.834876    6167 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:45:44.856146 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834927    6167 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834931    6167 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834934    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834937    6167 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834940    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834942    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834945    6167 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834948    6167 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834950    6167 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834953    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834955    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834958    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834961    6167 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834963    6167 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834966    6167 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834968    6167 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834971    6167 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834973    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834976    6167 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:45:44.856540 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834978    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834981    6167 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834984    6167 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834986    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834989    6167 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834991    6167 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834993    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834996    6167 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.834999    6167 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835001    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835004    6167 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835007    6167 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835009    6167 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835012    6167 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835020    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835022    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835025    6167 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835027    6167 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835030    6167 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835032    6167 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:45:44.857063 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835035    6167 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835038    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835040    6167 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835043    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835046    6167 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835048    6167 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835051    6167 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835053    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835057    6167 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835059    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835061    6167 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835064    6167 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835069    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835072    6167 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835074    6167 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835077    6167 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835079    6167 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835082    6167 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835084    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835087    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:45:44.857553 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835090    6167 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835092    6167 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835095    6167 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835097    6167 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835100    6167 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835102    6167 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835105    6167 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835115    6167 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835118    6167 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835121    6167 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835123    6167 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835126    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835130    6167 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835133    6167 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835136    6167 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835138    6167 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835141    6167 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835143    6167 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835146    6167 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:45:44.858071 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835149    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835151    6167 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835154    6167 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835157    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835159    6167 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835163    6167 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835166    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835169    6167 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835171    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835174    6167 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835177    6167 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835179    6167 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835182    6167 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835185    6167 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835187    6167 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835191    6167 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835195    6167 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835198    6167 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835200    6167 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:45:44.858591 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835204    6167 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835208    6167 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835217    6167 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835220    6167 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835223    6167 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835225    6167 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835229    6167 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835233    6167 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:44.835235    6167 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.835282    6167 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.836887    6167 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.836971    6167 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.837365    6167 server.go:1014] "Starting client certificate rotation"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.837595    6167 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.837820    6167 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:39:30 +0000 UTC" deadline="2028-07-01 07:40:23.714543728 +0000 UTC"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.837865    6167 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14533h54m38.87668039s"
Nov 03 17:45:44.859390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.838165    6167 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:45:44.860080 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.839256    6167 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:45:44.860080 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.840680    6167 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:45:44.860080 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.842729    6167 log.go:25] "Validated CRI v1 image API"
Nov 03 17:45:44.860080 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.843424    6167 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:45:44.860080 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.847262    6167 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9cd359b3-3316-45ae-af1a-4cbceff0a6ed:/dev/nvme0n1p4 fcf34f94-4228-4d45-8c3a-47b77eea6e2f:/dev/nvme0n1p3]
Nov 03 17:45:44.860245 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.847288    6167 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95/userdata/shm major:0 minor:489 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95/userdata/shm major:0 minor:474 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172/userdata/shm major:0 minor:455 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d/userdata/shm major:0 minor:389 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7937eb9a256b8eecceedb6a37a2e57b50d9269ee4b70b6475a390ade72203c81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7937eb9a256b8eecceedb6a37a2e57b50d9269ee4b70b6475a390ade72203c81/userdata/shm major:0 minor:487 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a83dcd85a67a063d980e6dc6f7f08905992bb16e94094130e3e0af6f378225fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a83dcd85a67a063d980e6dc6f7f08905992bb16e94094130e3e0af6f378225fa/userdata/shm major:0 minor:507 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b09440f78484f7bfc73778c34b13dcd3535e8374c6c8a200fb9c897f4dfcf298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b09440f78484f7bfc73778c34b13dcd3535e8374c6c8a200fb9c897f4dfcf298/userdata/shm major:0 minor:503 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b/userdata/shm major:0 minor:483 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6c626d2b96509e6222701aac7c8d0e2d1d28e8d572da2f403b8a3601653c26a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6c626d2b96509e6222701aac7c8d0e2d1d28e8d572da2f403b8a3601653c26a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e445f6815d1c366d2ee1fd82bad71272edaac6496102dc6f47fc11934510b77a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e445f6815d1c366d2ee1fd82bad71272edaac6496102dc6f47fc11934510b77a/userdata/shm major:0 minor:456 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b/userdata/shm major:0 minor:461 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a/userdata/shm major:0 minor:392 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b9dedb-2e8f-47d8-9a9f-3398edae5365/volumes/kubernetes.io~projected/kube-api-access-jwzbb:{mountpoint:/var/lib/kubelet/pods/03b9dedb-2e8f-47d8-9a9f-3398edae5365/volumes/kubernetes.io~projected/kube-api-access-jwzbb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volumes/kubernetes.io~projected/kube-api-access-z2vks:{mountpoint:/var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volumes/kubernetes.io~projected/kube-api-access-z2vks major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08a559aa-cebd-40a6-9393-3feb287029ee/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/08a559aa-cebd-40a6-9393-3feb287029ee/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:480 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5/volumes/kubernetes.io~projected/kube-api-access-4m7ds:{mountpoint:/var/lib/kubelet/pods/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5/volumes/kubernetes.io~projected/kube-api-access-4m7ds major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52/volumes/kubernetes.io~projected/kube-api-access-jpjm7:{mountpoint:/var/lib/kubelet/pods/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52/volumes/kubernetes.io~projected/kube-api-access-jpjm7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16fc5076-624a-47e0-9430-9550882714d8/volumes/kubernetes.io~projected/kube-api-access-p45jf:{mountpoint:/var/lib/kubelet/pods/16fc5076-624a-47e0-9430-9550882714d8/volumes/kubernetes.io~projected/kube-api-access-p45jf major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16fc5076-624a-47e0-9430-9550882714d8/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/16fc5076-624a-47e0-9430-9550882714d8/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:481 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/kube-api-access-79mh5:{mountpoint:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/kube-api-access-79mh5 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/registry-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce9cf1a-209e-489c-bae4-552a734e3e91/volumes/kubernetes.io~projected/kube-api-access-tdcj2:{mountpoint:/var/lib/kubelet/pods/1ce9cf1a-209e-489c-bae4-552a734e3e91/volumes/kubernetes.io~projected/kube-api-access-tdcj2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f389b73-c992-49ac-ba2d-cf69b91f2b1e/volumes/kubernetes.io~projected/kube-api-access-8cbdf:{mountpoint:/var/lib/kubelet/pods/1f389b73-c992-49ac-ba2d-cf69b91f2b1e/volumes/kubernetes.io~projected/kube-api-access-8cbdf major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f389b73-c992-49ac-ba2d-cf69b91f2b1e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1f389b73-c992-49ac-ba2d-cf69b91f2b1e/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/kube-api-access-w9h75:{mountpoint:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/kube-api-access-w9h75 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/registry-tls major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~projected/kube-api-access-wzxwt:{mountpoint:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~projected/kube-api-access-wzxwt major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/default-certificate major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/stats-auth major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7/volumes/kubernetes.io~projected/kube-api-access-jbv9j:{mountpoint:/var/lib/kubelet/pods/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7/volumes/kubernetes.io~projected/kube-api-access-jbv9j major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7/volumes/kubernetes.io~secret/signing-key major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd2bd81-c082-46e4-87c2-226f1acf6c07/volumes/kubernetes.io~projected/kube-api-access-5th4k:{mountpoint:/var/lib/kubelet/pods/2fd2bd81-c082-46e4-87c2-226f1acf6c07/volumes/kubernetes.io~projected/kube-api-access-5th4k major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36506792-bfe4-4a9f-b27c-c2096cb67ebe/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/36506792-bfe4-4a9f-b27c-c2096cb67ebe/volumes/kubernetes.io~secret/tls-certificates major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ae4fdcf-846d-464a-a6a5-28713517c9aa/volumes/kubernetes.io~projected/kube-api-access-9524c:{mountpoint:/var/lib/kubelet/pods/3ae4fdcf-846d-464a-a6a5-28713517c9aa/volumes/kubernetes.io~projected/kube-api-access-9524c major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ae4fdcf-846d-464a-a6a5-28713517c9aa/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3ae4fdcf-846d-464a-a6a5-28713517c9aa/volumes/kubernetes.io~secret/metrics-tls major:0 minor:482 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/kube-api-access-2fmrv:{mountpoint:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/kube-api-access-2fmrv major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/registry-tls major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4548c83e-c9e2-45fe-8ec0-3e450486cfd9/volumes/kubernetes.io~projected/kube-api-access-bxfpt:{mountpoint:/var/lib/kubelet/pods/4548c83e-c9e2-45fe-8ec0-3e450486cfd9/volumes/kubernetes.io~projected/kube-api-access-bxfpt major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4803ee-0fee-44d8-a5d9-7931656d6e7c/volumes/kubernetes.io~projected/kube-api-access-rk2rc:{mountpoint:/var/lib/kubelet/pods/4e4803ee-0fee-44d8-a5d9-7931656d6e7c/volumes/kubernetes.io~projected/kube-api-access-rk2rc major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50141ec0-76c0-453c-9ecc-e5c7c227dad2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/50141ec0-76c0-453c-9ecc-e5c7c227dad2/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~projected/kube-api-access-k9dnr:{mountpoint:/var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~projected/kube-api-access-k9dnr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bcba5ac-7a87-4aa9-b3e3-a474946ba957/volumes/kubernetes.io~projected/kube-api-access-cxjxd:{mountpoint:/var/lib/kubelet/pods/5bcba5ac-7a87-4aa9-b3e3-a474946ba957/volumes/kubernetes.io~projected/kube-api-access-cxjxd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63acd3e4-e973-4124-afcf-39f855f0e4aa/volumes/kubernetes.io~projected/kube-api-access-7kv8n:{mountpoint:/var/lib/kubelet/pods/63acd3e4-e973-4124-afcf-39f855f0e4aa/volumes/kubernetes.io~projected/kube-api-access-7kv8n major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69cbe72d-a018-4a89-8951-a25e50d6d33b/volumes/kubernetes.io~projected/kube-api-access-29nt9:{mountpoint:/var/lib/kubelet/pods/69cbe72d-a018-4a89-8951-a25e50d6d33b/volumes/kubernetes.io~projected/kube-api-access-29nt9 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69cbe72d-a018-4a89-8951-a25e50d6d33b/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/69cbe72d-a018-4a89-8951-a25e50d6d33b/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e77045a-8555-4a49-9ccd-47a14931763b/volumes/kubernetes.io~projected/kube-api-access-xnx8r:{mountpoint:/var/lib/kubelet/pods/7e77045a-8555-4a49-9ccd-47a14931763b/volumes/kubernetes.io~projected/kube-api-access-xnx8r major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911/volumes/kubernetes.io~projected/kube-api-access-f8vm2:{mountpoint:/var/lib/kubelet/pods/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911/volumes/kubernetes.io~projected/kube-api-access-f8vm2 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d645b31-5684-42a1-b305-59fe985149c4/volumes/kubernetes.io~projected/kube-api-access-b68wl:{mountpoint:/var/lib/kubelet/pods/8d645b31-5684-42a1-b305-59fe985149c4/volumes/kubernetes.io~projected/kube-api-access-b68wl major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/volumes/kubernetes.io~projected/kube-api-access-khf9l:{mountpoint:/var/lib/kubelet/pods/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/volumes/kubernetes.io~projected/kube-api-access-khf9l major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9392d9f2-6a5c-4b6a-b048-a20227a79836/volumes/kubernetes.io~projected/kube-api-access-5qbjl:{mountpoint:/var/lib/kubelet/pods/9392d9f2-6a5c-4b6a-b048-a20227a79836/volumes/kubernetes.io~projected/kube-api-access-5qbjl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d/volumes/kubernetes.io~projected/kube-api-access-82gxq:{mountpoint:/var/lib/kubelet/pods/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d/volumes/kubernetes.io~projected/kube-api-access-82gxq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9807c6c6-ef8b-4769-bf2e-97b5c5a23604/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9807c6c6-ef8b-4769-bf2e-97b5c5a23604/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:501 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ef120cf-f11d-435c-96c4-96d73f511afa/volumes/kubernetes.io~projected/kube-api-access-tr87j:{mountpoint:/var/lib/kubelet/pods/9ef120cf-f11d-435c-96c4-96d73f511afa/volumes/kubernetes.io~projected/kube-api-access-tr87j major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ef120cf-f11d-435c-96c4-96d73f511afa/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9ef120cf-f11d-435c-96c4-96d73f511afa/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad0fe61-4d34-46c0-9b9f-263eb91dc103/volumes/kubernetes.io~projected/kube-api-access-mblft:{mountpoint:/var/lib/kubelet/pods/aad0fe61-4d34-46c0-9b9f-263eb91dc103/volumes/kubernetes.io~projected/kube-api-access-mblft major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad0fe61-4d34-46c0-9b9f-263eb91dc103/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/aad0fe61-4d34-46c0-9b9f-263eb91dc103/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9984e6b-2191-4627-96ac-9d0b53762a46/volumes/kubernetes.io~projected/kube-api-access-4c4jh:{mountpoint:/var/lib/kubelet/pods/b9984e6b-2191-4627-96ac-9d0b53762a46/volumes/kubernetes.io~projected/kube-api-access-4c4jh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bae3db47-36e8-4957-9c10-fd361815db36/volumes/kubernetes.io~projected/kube-api-access-zpctd:{mountpoint:/var/lib/kubelet/pods/bae3db47-36e8-4957-9c10-fd361815db36/volumes/kubernetes.io~projected/kube-api-access-zpctd major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bae3db47-36e8-4957-9c10-fd361815db36/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bae3db47-36e8-4957-9c10-fd361815db36/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5703035-ada3-4cfb-b77b-4a8e9bcbe063/volumes/kubernetes.io~projected/kube-api-access-2nmwq:{mountpoint:/var/lib/kubelet/pods/d5703035-ada3-4cfb-b77b-4a8e9bcbe063/volumes/kubernetes.io~projected/kube-api-access-2nmwq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5703035-ada3-4cfb-b77b-4a8e9bcbe063/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d5703035-ada3-4cfb-b77b-4a8e9bcbe063/volumes/kubernetes.io~secret/metrics-certs major:0 minor:502 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef95306f-a39f-419e-84d7-6496c754701b/volumes/kubernetes.io~projected/kube-api-access-k6xrz:{mountpoint:/var/lib/kubelet/pods/ef95306f-a39f-419e-84d7-6496c754701b/volumes/kubernetes.io~projected/kube-api-access-k6xrz major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~projected/kube-api-access-px9k7:{mountpoint:/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~projected/kube-api-access-px9k7 major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00/volumes/kubernetes.io~projected/kube-api-access-p8hxz:{mountpoint:/var/lib/kubelet/pods/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00/volumes/kubernetes.io~projected/kube-api-access-p8hxz major:0 minor:335 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ef1a9d8b95e3b0c635c00beba14c6c905bcd32a2b2da89041fca2bceaf64283f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d6fcb4ffd6003ae3cbe71d05a2ef29a8a0624fcf8476a69c72080dc9f6b2ea6d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c9ed9f07fbc6b5075388b2ab8538310237f66dc1dc43744e1008acaa5e718ab9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/56b99039f81cf22cacb2d5da92b983eec36cc5526a3920cc86401fe73b0a81c3/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fbbf5669f2880648b6408095589ca2c61dfeebe0c67ad2252dc7a56bea045290/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b492cf4fc43d2c84b16c0792fbf4721f12d27dd66318431152b9c251d42ed8b4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3005258ca74ec7ee6adbd8873f116119597aa8dbfbaf1a460fc6f112a3200e9b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5ca9400c1362cf76d87e3539e9d709057db83f2a78f845aa89356767df786847/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/ba57a03055c18ff20981d30f68f6e063c5470dac8efdb18a6e92822f94ea79d1/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/e3c744eb34585aa88de3ea716ca7a94ac84bad8f793681d1dd3e90a69dc3326e/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/dbdebc26fb7f0487fc0d7c02bdc0f50c2c052434932723bf414be7d6596c6cd9/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/b6a6f8bb7acbb012d6fa3ce28e9dca9df1386f12f6bff8d459906d555cfafe47/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6c0aa472c895df569d64a97238a6919cf14b7a019c7d4fc6565b5a4bab14a4d2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4eabba22e645b7756b236efa6e05af8b23251296ef85512f97be522f79e37af9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bf592a88a33ea9cf4e103f464ba46342fcceba5b1bac9632649389b91bf938a4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4be328173c484afd1bc64f3d65fc39eba5e40537417153b60011a743507297b5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/82d2c879b4e4156ac8a55dc3cc7fe7be8b7065f7ba33f131ae69d0f2d28697b2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/340f061fdfca800972e97a5cd7450c37aca3c65c2c6b000f43bc4ac7392f6b11/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d06dd731fe8a350d7138f50edc7d265c6445fa3fe029dd29276ef1ce01203474/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1619dce614fbaebdc5598ee66103634dbe065e399f17aa8c8538c81bb5aa3f44/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/46eed2e6a6e6ae6b475d4d02c712ef934da08b04d58cca10848c18b16c459101/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/a25a3cfdcbcbdf677619c7874c89b9b72913614d14da59b38eb65756aeb3f3c4/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2f0a85ce315b1f470338f9a3bccccaa2585d202b3d4f9687be3e17124233b0bf/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/288e13161ea7417ebe9a1c26d8f3d6b36ad2915b1551d1106cb4243fd3005200/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/53d59c08a6c2a7cb3a9d50695dbfd952f6e187e15fc789372e70de36222cbe26/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/18b74341cef1a2f0bc87ae1de126007f80f0b6ad1fdc1c62d0f6c3c4c3a9607b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/432c08bc78de4d644a12c00da7f23b1597dd4d9addafe5979828f8aabef86219/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b4ba0d6368e27bfa151fe9948d723a1bb4265b8c865ebdc0a83ff904b3d1d623/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/e8d32c5633f2d9a2726bf41c27f468f09c63416e976d3a55ee0b89c2df7923a4/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/adc8df41c297f42a50217d167433bf1e9de8bf37e8e7cc898a97be0ea1dcb00b/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/58278d3f0af9cac9cd6de4ca8a578b276c21e00a982b6eab2610d26da3054606/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/38920ffac1da5879ea05d7355ac42f1b4007d02227e1ec0c0e9bc0b8a30710ed/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/db79a423252ab6e3e06e19fb550ef6e7830cdab1c12d4346e4ee8e97f12343a5/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9b0e6b72d52e053e89118cd6eb5d3ba27fe73195cbbb1a08299938e3dc048de2/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/b89ee58a8f26e609f0280dc96888f6298d5a484e643f2f74c59abbfcb2545ef0/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/e6973eca4ed5433d7b3c8bbb59f5626a529d76133df1d94b791005d91123d6c2/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/639f0fbb8a0936d966f10a86f2fcc950f0bd98fdf9a00ce6cb91a96f487c98d3/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/5d8902be2cfe999ab449be36a8a202836b5a75606e37e79cf0da3f3003a09fbd/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/b8dcd103210d18678c4319abc2ce1faaba9f1d2aa1f7d66cc887d88b759f398c/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/9066b2b68916eb9f347ef58c2d36907cc2afcc8a020e91b4591e9a0f67947e5a/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/01884a7f9b6964b657bf02d57220d56e691409439afcf4c1332a120257c65669/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/c8e7b19e24f510b22d71fa403bb988db47932d616c60632450e9a7c71e6782b7/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/91df0afee9e433aa406cb2b2a461ec49eda5aa6944b2921947a645ec16336335/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/0f4e3aade7e04d08a77b2509f4d868ab7615606ee006bf5bb4b6f3389c8fabe4/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/39aace49de8ce673ea77b20fa96d123ee2d8801e7c88584794943135cd39fa4f/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/efaa96c30d0504cdce31fb33bb32348dcdb918ea21d34c5b4b742b5f44c222ee/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/a3ec85491c60936e2d6e02d4797471817a183acd6258eda4e03012e4d87ba054/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/2feb6c5a33cf6ef0a0bfa5962a3ac66a222c98086a48fd8aa28684d324e2c0c2/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/158d8f94fa189aca1d819ef7df3a41d962650cc2f86ecf5a88b84849addfae29/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/6ab3d7d241c19186623c3f7f0c1497cb36a727cac9e71fab17a932c69b826668/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/b9322cfcd5d0d110fc28b512fe5bdc4650572b04c09375fe26a3123a981c71b0/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/76bb0120d0b307e8b9aaa848fe226d97be6f6290ac4c7c579019d8f328337329/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/8648b3073f9a114af077fb0201737b5a5bea9bca62d69ee668070b78b0aa07ff/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/f864a1aee49c4d61be32547d54f72bbf4e3672a7210bbe5ebd86a9a0da8e96e8/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/5f13c936809f2efe8445bf1e846f1b2e4310a9d9823476724a07a6f0fb9cf25b/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/a78ffe3f5144f7dfdbfd627e991a7abe70550dd9dbe47dde13ba3a63416bcac1/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/1c685ea233dc09bc0cadf701baf513bf9e96bcbcbddf28f11539b54132a4180e/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/9731cb1331161bee522e3043ac5ab2d174fadf3acdb5ec69c35686aa0b0351b0/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/e02f0e67434908eb75c7ab6909d22e5f6d287c309d722256cd1726b9be2ae6b3/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/446cd5ba3ef28c1a151731e9e475b3e80886b154e7e8d8c84a0324703f482c66/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/d670c4f75b7d45259e86bb7573acf782e398dc1cd7d935e7b42685982e05c43e/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/60f1f33181bbb865f2297f1d8fb1143ca039d41338ee88778127d40b82ec378d/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/3d6e20659344309c1b032771e95245406e7a9a4ca515334258753c1e5d4ffdea/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/336d000505ea8756894a28e53a575ed67ddfb59ba6d14c469bcd5a21cbc2b786/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/2c0971db08d5aa55dbbcd54ad8f3c1c6a4e3e832a982c889767deaa3cc897e56/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/ac96ebd89baad4ebdf1747073d0f227feea725d09ec9aadc2e7d897ca76e71b8/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/a80c4731677a8d74c71fdf07613d008ff2f431a1e889be3f3815871e07dc2de0/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/016ab7e519ec6007db53d3321fe20d0e776f5303f2b9800a0daa030fa374576a/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/c44987c01aeec30c344c98bd7a4a21d0894e248141beee1c451b8e645d815bea/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/cd404fbfaa3e8ac42d49f68e0a46059d6458f7aa4780a66109b9c2151e62e421/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/a162e62eeacf697e88a889a8fecc152f0f7058ca4ffc3c620c8ac43f558509fb/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/ec49b00f3a877d709ce61c08a611d2e67cebf91db5afb462e6c265538b3b8d66/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/6e2342a4da51fa11699dd227634e1a2df64d6d5919191d751110d015bb8c6a3c/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/42aa67e0cd1ed36fa8eafa1cc87c9b0789727c3808656fb9aeda59b09a847966/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/cd543fb8f2b3bee71df8dd509b7d1daf2b50e35a87b6f883d8d4e443eec6dcde/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-517:{mountpoint:/var/lib/containers/storage/overlay/9c57330259416286dcf5120cace0eadb2929c20e18e739530b0f08fbdf1b6654/merged major:0 minor:517 fsType:overlay blockSize:0} overlay_0-519:{mountpoint:/var/lib/containers/storage/overlay/f12cee63c4cbf91f4dc64149c11d27dff74769455a2130b93c9194a7c2420e57/merged major:0 minor:519 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/bb1fbeece2f6d3c5b1af87aef882fdc88005e20ae04993469e3d71c250a3d14d/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/6d656da172d3cbba9eb91697a34bbff30d9c77ac337e39bddba414b66e10a501/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/efdaaebf2439121685c5d006838308854088b3221fbc3750aba1f0f10d2dde7d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-535:{mountpoint:/var/lib/containers/storage/overlay/3b718c5072b23dfd5d8b78700d7f2eefdadb301ad30b65f80ffcd943fb9ea0b5/merged major:0 minor:535 fsType:overlay blockSize:0} overlay_0-540:{mountpoint:/var/lib/containers/storage/overlay/bf4f3ea5925e5279a678e78fc7999583e4a7db06b02a9470f7ff51a81fbbcd72/merged major:0 minor:540 fsType:overlay blockSize:0} overlay_0-546:{mountpoint:/var/lib/containers/storage/overlay/281e3c14892dee47bee411b408bcd47465592065fd2bb39f2e557b55d0267f34/merged major:0 minor:546 fsType:overlay blockSize:0} overlay_0-552:{mountpoint:/var/lib/containers/storage/overlay/29b62374607d30968e4d5d821c2bc760039c90633fb0039704c97efbe9ba005f/merged major:0 minor:552 fsType:overlay blockSize:0} overlay_0-557:{mountpoint:/var/lib/containers/storage/overlay/2929a76cffc0abd1db7c9ea361618080e3a534c9c2fa59a76e1f83da087a3cb5/merged major:0 minor:557 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/73a60e99fe7b33ff4497cc06a7d17b496b121b322210dcfeff2b9a659483246f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c8f3797e0ec91194f00c76b0a5389e1f4118e70594cc4eb817f4fc15133dd886/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/dbf11118a587d208a12c706f080ea44953a9659f15b30effa9fd5575defe7911/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c8bd6852af9839075b7cd1e0272c80646f7d429974fd697b810d833d17f4d6fc/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c8898ebfd78f2079175d93f49c5385f3d3f7fac43bb512d0b2d22e13f160cfbf/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3b6603f8094f45a0bcc3ce17154cd1f17c97c4a557db7d8187bb3a8e3eaf46fd/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a0336210f43e8a7ee163c1d2be791f6c8e2c48fe7b968ff2964de2242e296364/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/971aad50bdae4dfa89c9612dc0a8ade41a804f8ed162486089da156f4f1b0ea5/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 17:45:44.862357 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.861859    6167 manager.go:217] Machine: {Timestamp:2026-11-03 17:45:44.861342991 +0000 UTC m=+0.091668057 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105806 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2267048449c1d682eed69ee08bc1b5 SystemUUID:ec226704-8449-c1d6-82ee-d69ee08bc1b5 BootID:326d59a0-c003-4215-8a45-21d902b1395a Filesystems:[{Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:452 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a337bcaed53affb26ae6cc00532c598a067ecac0727d1528a45747351032f95/userdata/shm DeviceMajor:0 DeviceMinor:489 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-540 DeviceMajor:0 DeviceMinor:540 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00/volumes/kubernetes.io~projected/kube-api-access-p8hxz DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:453 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16fc5076-624a-47e0-9430-9550882714d8/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:481 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df9fa30943d8c8a61dfe1ef457a0c0280279b09c1d1e563b58ae825c70aab834/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bae3db47-36e8-4957-9c10-fd361815db36/volumes/kubernetes.io~projected/kube-api-access-zpctd DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9ef120cf-f11d-435c-96c4-96d73f511afa/volumes/kubernetes.io~projected/kube-api-access-tr87j DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28a7a40641617fcd533940a15bba4dc2c9ab9f35d1e4e23b97dbddaa53062f07/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/36506792-bfe4-4a9f-b27c-c2096cb67ebe/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:446 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:463 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ae4fdcf-846d-464a-a6a5-28713517c9aa/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:482 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78830bd9187e384daf772801f37de5bd89728b6b49513db6959ce3727e31406e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8d645b31-5684-42a1-b305-59fe985149c4/volumes/kubernetes.io~projected/kube-api-access-b68wl DeviceMajor:0 DeviceMinor:274 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/9392d9f2-6a5c-4b6a-b048-a20227a79836/volumes/kubernetes.io~projected/kube-api-access-5qbjl DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce9cf1a-209e-489c-bae4-552a734e3e91/volumes/kubernetes.io~projected/kube-api-access-tdcj2 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/623cd02c06f6564082da05bcc657ac88e6e88932e41641c31d37b18f054bd11e/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aad0fe61-4d34-46c0-9b9f-263eb91dc103/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da21e8959b46dfa476603689b1e02644a30becda9fec4b9e52d7e848a6db284b/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7/volumes/kubernetes.io~projected/kube-api-access-jbv9j DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e618f9c5b838b29e87f01b4e1123c583924697f55dcb386e1ef732f1aa98f15b/userdata/shm DeviceMajor:0 DeviceMinor:461 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f389b73-c992-49ac-ba2d-cf69b91f2b1e/volumes/kubernetes.io~projected/kube-api-access-8cbdf DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63acd3e4-e973-4124-afcf-39f855f0e4aa/volumes/kubernetes.io~projected/kube-api-access-7kv8n DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-546 DeviceMajor:0 DeviceMinor:546 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5721823413c91080f9ba653a28645d309b2c63c53e347c402949f113f6c66df7/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f389b73-c992-49ac-ba2d-cf69b91f2b1e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911/volumes/kubernetes.io~projected/kube-api-access-f8vm2 DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/16fc5076-624a-47e0-9430-9550882714d8/volumes/kubernetes.io~projected/kube-api-access-p45jf DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-517 DeviceMajor:0 DeviceMinor:517 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f76c6110ff9c78efa7709505ca2f3be727571470f81b40a6e2c849a358ae1ce/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/942c89dd818745d8cee2a1bb5760c51c9beab40ebb3119d65df213915110e34d/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aad0fe61-4d34-46c0-9b9f-263eb91dc103/volumes/kubernetes.io~projected/kube-api-access-mblft DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-519 DeviceMajor:0 DeviceMinor:519 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b872afa2d338ef5cce8565b28ca221d591a99339c482220cf0fd4e5699b58a6d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9984e6b-2191-4627-96ac-9d0b53762a46/volumes/kubernetes.io~projected/kube-api-access-4c4jh DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0429e44b9bfd7b0a494c00cd2b8da58597898d0e228c87a2b17e88c3ac1111d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9154fd1073ef3e0a70c6f5515321656a9a4eabdc74ed3f8f7f095d9bea9ef65/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/469036a2b7d65333a720fb1b0345c7c35d6bef54bec4c06f5fc5e3713f66217b/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d87cd62b741a2a2e03d0a2618144e8085ba9a3c2523cd376a15a1be0d8d14c83/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:459 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:388 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:391 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-552 DeviceMajor:0 DeviceMinor:552 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e77045a-8555-4a49-9ccd-47a14931763b/volumes/kubernetes.io~projected/kube-api-access-xnx8r DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/08a559aa-cebd-40a6-9393-3feb287029ee/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:480 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~projected/kube-api-access-k9dnr DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d5703035-ada3-4cfb-b77b-4a8e9bcbe063/volumes/kubernetes.io~projected/kube-api-access-2nmwq DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d/userdata/shm DeviceMajor:0 DeviceMinor:389 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5054e8585353f9d71425ba37a72e3bb5a23a4aa6aac3171c74e37f396d9ca5b/userdata/shm DeviceMajor:0 DeviceMinor:483 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c/volumes/kubernetes.io~projected/kube-api-access-khf9l DeviceMajor:0 DeviceMinor:237 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a83dcd85a67a063d980e6dc6f7f08905992bb16e94094130e3e0af6f378225fa/userdata/shm DeviceMajor:0 DeviceMinor:507 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50141ec0-76c0-453c-9ecc-e5c7c227dad2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1a652a-2b2f-412a-bf99-3fab0319dd38/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6c626d2b96509e6222701aac7c8d0e2d1d28e8d572da2f403b8a3601653c26a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5/volumes/kubernetes.io~projected/kube-api-access-4m7ds DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes/kubernetes.io~projected/kube-api-access-w9h75 DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b09440f78484f7bfc73778c34b13dcd3535e8374c6c8a200fb9c897f4dfcf298/userdata/shm DeviceMajor:0 DeviceMinor:503 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dc08f48730e24e20382370a6a66dd9e0a16ab8b588b6cbc6fdd2210d182f048/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52/volumes/kubernetes.io~projected/kube-api-access-jpjm7 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/69cbe72d-a018-4a89-8951-a25e50d6d33b/volumes/kubernetes.io~projected/kube-api-access-29nt9 DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e445f6815d1c366d2ee1fd82bad71272edaac6496102dc6f47fc11934510b77a/userdata/shm DeviceMajor:0 DeviceMinor:456 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/kube-api-access-2fmrv DeviceMajor:0 DeviceMinor:460 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4803ee-0fee-44d8-a5d9-7931656d6e7c/volumes/kubernetes.io~projected/kube-api-access-rk2rc DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18182a13f4e47a5cea0a994c3d66f1507129ed1e9cd2dabb7cc6099c7adc712a/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~projected/kube-api-access-px9k7 DeviceMajor:0 DeviceMinor:469 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9aac95a4c8f5b162b7a5bd386ab46febd26cd256e23f922f4e40c7aa3d14cd8/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0e6e1251bc03faf538157ce79bb2a3ea27c1cfc72d4a79ba8955353db1fe663/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95/userdata/shm DeviceMajor:0 DeviceMinor:474 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7937eb9a256b8eecceedb6a37a2e57b50d9269ee4b70b6475a390ade72203c81/userdata/shm DeviceMajor:0 DeviceMinor:487 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-535 DeviceMajor:0 DeviceMinor:535 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4548c83e-c9e2-45fe-8ec0-3e450486cfd9/volumes/kubernetes.io~projected/kube-api-access-bxfpt DeviceMajor:0 DeviceMinor:451 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:464 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5703035-ada3-4cfb-b77b-4a8e9bcbe063/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:502 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-557 DeviceMajor:0 DeviceMinor:557 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffadc8f475818fab870229d9c444616cb266a68928c72bb210294f431414682d/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4357c52c-00a5-4bb2-bf8e-315d3760e129/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9807c6c6-ef8b-4769-bf2e-97b5c5a23604/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:501 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ef120cf-f11d-435c-96c4-96d73f511afa/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdffee9d0a7290ae550775d4a12e582c5f5278f43f65d7f3544e2e58f7164c95/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bcba5ac-7a87-4aa9-b3e3-a474946ba957/volumes/kubernetes.io~projected/kube-api-access-cxjxd DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/07cd964f-0f58-48bb-a2b5-81b1d0f355b2/volumes/kubernetes.io~projected/kube-api-access-z2vks DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bae3db47-36e8-4957-9c10-fd361815db36/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6f00fec8bb863e3d19b8e2d3d7b6289b7e8a9ce76f972d5432abf2b70d272c/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91eef3162b75ee720e533be2bf20f20178c7468e053f4b4dc24768dcd0e8f89c/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eecd42f8c826ea97fe2e8951dab87ae84dc0c2d2b28a2dfd09c9948223d7fe4b/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1d3a57b0266f5643a81b47944bb32c799bcf5d8e1d52a6f2f522c668201d44a/userdata/shm DeviceMajor:0 DeviceMinor:392 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd2bd81-c082-46e4-87c2-226f1acf6c07/volumes/kubernetes.io~projected/kube-api-access-5th4k DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ae4fdcf-846d-464a-a6a5-28713517c9aa/volumes/kubernetes.io~projected/kube-api-access-9524c DeviceMajor:0 DeviceMinor:239 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/03b9dedb-2e8f-47d8-9a9f-3398edae5365/volumes/kubernetes.io~projected/kube-api-access-jwzbb DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/kube-api-access-79mh5 DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d/volumes/kubernetes.io~projected/kube-api-access-82gxq DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e45fafa81a6c449e4be20c06ab3ae15defd0ef78509825412106e079191833ac/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ef95306f-a39f-419e-84d7-6496c754701b/volumes/kubernetes.io~projected/kube-api-access-k6xrz DeviceMajor:0 DeviceMinor:334 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/69cbe72d-a018-4a89-8951-a25e50d6d33b/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2947ed62-ecb7-4ccd-935b-b1870cffb3e5/volumes/kubernetes.io~projected/kube-api-access-wzxwt DeviceMajor:0 DeviceMinor:238 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5759ab83c745badec73f9af566a03edae15ca888c32cb69d586695e0b4a83172/userdata/shm DeviceMajor:0 DeviceMinor:455 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:18182a13f4e47a5 MacAddress:0e:19:ec:3c:d6:e6 Speed:10000 Mtu:8901} {Name:1ce03a1b15a00f9 MacAddress:6e:a4:a7:bf:d5:e0 Speed:10000 Mtu:8901} {Name:28a7a40641617fc MacAddress:36:8a:f7:34:46:19 Speed:10000 Mtu:8901} {Name:3a337bcaed53aff MacAddress:92:0d:b3:a2:23:cb Speed:10000 Mtu:8901} {Name:3d0a9758e7075cb MacAddress:9a:c5:58:28:8c:ce Speed:10000 Mtu:8901} {Name:3e6f00fec8bb863 MacAddress:a6:87:93:01:91:55 Speed:10000 Mtu:8901} {Name:5759ab83c745bad MacAddress:2a:b9:62:76:e8:a2 Speed:10000 Mtu:8901} {Name:58d215ead268604 MacAddress:0e:0e:f4:07:86:3d Speed:10000 Mtu:8901} {Name:623cd02c06f6564 MacAddress:5a:8a:2a:a4:1c:df Speed:10000 Mtu:8901} {Name:7937eb9a256b8ee MacAddress:7a:d5:16:f2:37:a8 Speed:10000 Mtu:8901} {Name:91eef3162b75ee7 MacAddress:ca:e3:ff:73:5a:c6 Speed:10000 Mtu:8901} {Name:942c89dd818745d MacAddress:96:48:44:c1:f9:59 Speed:10000 Mtu:8901} {Name:a83dcd85a67a063 MacAddress:12:fb:b8:d3:fd:53 Speed:10000 Mtu:8901} {Name:b09440f78484f7b MacAddress:9a:da:72:5d:db:01 Speed:10000 Mtu:8901} {Name:b872afa2d338ef5 MacAddress:5a:67:ac:75:38:fd Speed:10000 Mtu:8901} {Name:b9aac95a4c8f5b1 MacAddress:8e:7d:ad:39:3a:84 Speed:10000 Mtu:8901} {Name:bdffee9d0a7290a MacAddress:06:ca:25:49:6d:b1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d4:f0:36:09 Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:e3:28:2e:9b:e5 Speed:0 Mtu:8901} {Name:c0e6e1251bc03fa MacAddress:f6:45:42:2f:8e:d1 Speed:10000 Mtu:8901} {Name:c5054e8585353f9 MacAddress:7e:78:98:de:2e:f7 Speed:10000 Mtu:8901} {Name:d87cd62b741a2a2 MacAddress:5e:0a:c9:3e:6b:82 Speed:10000 Mtu:8901} {Name:da21e8959b46dfa MacAddress:76:1b:da:55:37:70 Speed:10000 Mtu:8901} {Name:e445f6815d1c366 MacAddress:36:6b:fa:a9:01:23 Speed:10000 Mtu:8901} {Name:e45fafa81a6c449 MacAddress:72:1b:45:bf:cf:8a Speed:10000 Mtu:8901} {Name:e618f9c5b838b29 MacAddress:ce:ce:0f:7d:6e:9d Speed:10000 Mtu:8901} {Name:eecd42f8c826ea9 MacAddress:82:fe:8b:97:18:cd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d4:f0:36:09 Speed:0 Mtu:9001} {Name:f1d3a57b0266f56 MacAddress:16:6e:70:14:91:4a Speed:10000 Mtu:8901} {Name:f9154fd1073ef3e MacAddress:16:67:da:29:2c:f1 Speed:10000 Mtu:8901} {Name:fa2a748e2bdb6ea MacAddress:c2:6a:0a:f4:02:ca Speed:10000 Mtu:8901} {Name:ffadc8f475818fa MacAddress:d6:5f:3f:3e:cf:3d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:3a:5d:3a:cc:15:55 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:ba:5f:67:4a:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:45:44.862660 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862362    6167 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:45:44.862660 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862408    6167 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:45:44.862798 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862769    6167 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:45:44.862923 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862799    6167 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-108.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:45:44.862965 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862925    6167 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:45:44.862965 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862934    6167 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:45:44.862965 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862944    6167 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:45:44.862965 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.862958    6167 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:45:44.863074 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863051    6167 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:45:44.863240 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863230    6167 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:45:44.863345 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863336    6167 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:45:44.863383 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863374    6167 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:45:44.863415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863393    6167 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:45:44.863415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863404    6167 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:45:44.863415 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863412    6167 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:45:44.863530 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.863420    6167 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:45:44.864618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.864601    6167 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:45:44.864618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.864619    6167 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:45:44.865181 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.865163    6167 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:45:44.877219 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877203    6167 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:45:44.877219 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877218    6167 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:45:44.877343 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877333    6167 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 17:45:44.877527 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877516    6167 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:45:44.877564 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877544    6167 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:45:44.877657 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877648    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:45:44.877687 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877668    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:45:44.877687 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877677    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:45:44.877687 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877685    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877692    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877698    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877704    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877710    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877731    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877739    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877747    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877757    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877778    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:45:44.877808 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.877783    6167 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:45:44.881357 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.881336    6167 server.go:1263] "Started kubelet"
Nov 03 17:45:44.881445 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.881389    6167 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:45:44.881560 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.881489    6167 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:45:44.881624 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.881589    6167 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:45:44.881912 ip-10-0-141-108 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:45:44.882636 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.882440    6167 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:45:44.883337 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.883322    6167 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:45:44.888820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.888462    6167 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:45:44.888820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.888582    6167 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:45:44.888820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.888596    6167 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:45:44.888820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.888692    6167 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:45:44.889507 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:44.888952    6167 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:45:44.889948 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.889925    6167 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:45:44.890045 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.889985    6167 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:32 +0000 UTC" deadline="2028-07-20 18:38:08.877643358 +0000 UTC"
Nov 03 17:45:44.890045 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.890005    6167 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15000h52m23.987641758s"
Nov 03 17:45:44.891458 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891431    6167 factory.go:55] Registering systemd factory
Nov 03 17:45:44.891458 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891458    6167 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:45:44.891877 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891863    6167 factory.go:153] Registering CRI-O factory
Nov 03 17:45:44.891877 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891878    6167 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:45:44.891987 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891927    6167 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:45:44.891987 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891947    6167 factory.go:103] Registering Raw factory
Nov 03 17:45:44.891987 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.891961    6167 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:45:44.892357 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892345    6167 manager.go:319] Starting recovery of all containers
Nov 03 17:45:44.892922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892885    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16fc5076-624a-47e0-9430-9550882714d8" volumeName="kubernetes.io/projected/16fc5076-624a-47e0-9430-9550882714d8-kube-api-access-p45jf" seLinuxMountContext=""
Nov 03 17:45:44.893002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892925    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7" volumeName="kubernetes.io/projected/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-kube-api-access-jbv9j" seLinuxMountContext=""
Nov 03 17:45:44.893002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892940    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" volumeName="kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 17:45:44.893002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892953    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69cbe72d-a018-4a89-8951-a25e50d6d33b" volumeName="kubernetes.io/projected/69cbe72d-a018-4a89-8951-a25e50d6d33b-kube-api-access-29nt9" seLinuxMountContext=""
Nov 03 17:45:44.893002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892968    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94ef9fc0-8b2b-440a-bdd1-5684a5bc573d" volumeName="kubernetes.io/configmap/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-serviceca" seLinuxMountContext=""
Nov 03 17:45:44.893002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892980    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9984e6b-2191-4627-96ac-9d0b53762a46" volumeName="kubernetes.io/projected/b9984e6b-2191-4627-96ac-9d0b53762a46-kube-api-access-4c4jh" seLinuxMountContext=""
Nov 03 17:45:44.893002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.892993    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5703035-ada3-4cfb-b77b-4a8e9bcbe063" volumeName="kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893006    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce9cf1a-209e-489c-bae4-552a734e3e91" volumeName="kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893020    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b9dedb-2e8f-47d8-9a9f-3398edae5365" volumeName="kubernetes.io/projected/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kube-api-access-jwzbb" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893034    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52" volumeName="kubernetes.io/secret/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-serving-cert" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893047    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2947ed62-ecb7-4ccd-935b-b1870cffb3e5" volumeName="kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-stats-auth" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893060    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" volumeName="kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893072    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="904ebe0f-c56d-4bd5-aae8-53dd560c8f5c" volumeName="kubernetes.io/secret/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-cert" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893084    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893099    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36506792-bfe4-4a9f-b27c-c2096cb67ebe" volumeName="kubernetes.io/secret/36506792-bfe4-4a9f-b27c-c2096cb67ebe-tls-certificates" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893122    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae4fdcf-846d-464a-a6a5-28713517c9aa" volumeName="kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893135    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4548c83e-c9e2-45fe-8ec0-3e450486cfd9" volumeName="kubernetes.io/empty-dir/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-tmp" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893148    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef95306f-a39f-419e-84d7-6496c754701b" volumeName="kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893159    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" volumeName="kubernetes.io/secret/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-serving-cert" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893170    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2947ed62-ecb7-4ccd-935b-b1870cffb3e5" volumeName="kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893181    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7" volumeName="kubernetes.io/secret/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-key" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893192    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bae3db47-36e8-4957-9c10-fd361815db36" volumeName="kubernetes.io/projected/bae3db47-36e8-4957-9c10-fd361815db36-kube-api-access-zpctd" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893204    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893217    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" volumeName="kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-config" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893229    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893241    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00" volumeName="kubernetes.io/projected/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00-kube-api-access-p8hxz" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893253    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-bound-sa-token" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893265    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef120cf-f11d-435c-96c4-96d73f511afa" volumeName="kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893298    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a559aa-cebd-40a6-9393-3feb287029ee" volumeName="kubernetes.io/configmap/08a559aa-cebd-40a6-9393-3feb287029ee-nginx-conf" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893306    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52" volumeName="kubernetes.io/configmap/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-config" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893316    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2947ed62-ecb7-4ccd-935b-b1870cffb3e5" volumeName="kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle" seLinuxMountContext=""
Nov 03 17:45:44.893313 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893328    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef120cf-f11d-435c-96c4-96d73f511afa" volumeName="kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-service-ca-bundle" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893341    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad0fe61-4d34-46c0-9b9f-263eb91dc103" volumeName="kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893349    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893357    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/empty-dir/4357c52c-00a5-4bb2-bf8e-315d3760e129-ca-trust-extracted" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893366    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-kube-api-access-2fmrv" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893373    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9984e6b-2191-4627-96ac-9d0b53762a46" volumeName="kubernetes.io/empty-dir/b9984e6b-2191-4627-96ac-9d0b53762a46-tmp-dir" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893381    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bae3db47-36e8-4957-9c10-fd361815db36" volumeName="kubernetes.io/configmap/bae3db47-36e8-4957-9c10-fd361815db36-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893389    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef120cf-f11d-435c-96c4-96d73f511afa" volumeName="kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-tmp" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893400    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef95306f-a39f-419e-84d7-6496c754701b" volumeName="kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893411    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" volumeName="kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-trusted-ca" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893420    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d645b31-5684-42a1-b305-59fe985149c4" volumeName="kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893428    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5" volumeName="kubernetes.io/projected/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5-kube-api-access-4m7ds" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893435    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae4fdcf-846d-464a-a6a5-28713517c9aa" volumeName="kubernetes.io/empty-dir/3ae4fdcf-846d-464a-a6a5-28713517c9aa-tmp-dir" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893442    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4548c83e-c9e2-45fe-8ec0-3e450486cfd9" volumeName="kubernetes.io/projected/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-kube-api-access-bxfpt" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893450    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1a652a-2b2f-412a-bf99-3fab0319dd38" volumeName="kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-tuned" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893459    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e77045a-8555-4a49-9ccd-47a14931763b" volumeName="kubernetes.io/configmap/7e77045a-8555-4a49-9ccd-47a14931763b-iptables-alerter-script" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893467    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16fc5076-624a-47e0-9430-9550882714d8" volumeName="kubernetes.io/configmap/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893474    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893486    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893495    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd2bd81-c082-46e4-87c2-226f1acf6c07" volumeName="kubernetes.io/projected/2fd2bd81-c082-46e4-87c2-226f1acf6c07-kube-api-access-5th4k" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893502    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-installation-pull-secrets" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893511    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893519    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69cbe72d-a018-4a89-8951-a25e50d6d33b" volumeName="kubernetes.io/configmap/69cbe72d-a018-4a89-8951-a25e50d6d33b-telemetry-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893527    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad0fe61-4d34-46c0-9b9f-263eb91dc103" volumeName="kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893535    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07cd964f-0f58-48bb-a2b5-81b1d0f355b2" volumeName="kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-env-overrides" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893577    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893586    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2947ed62-ecb7-4ccd-935b-b1870cffb3e5" volumeName="kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-default-certificate" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893594    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4803ee-0fee-44d8-a5d9-7931656d6e7c" volumeName="kubernetes.io/projected/4e4803ee-0fee-44d8-a5d9-7931656d6e7c-kube-api-access-rk2rc" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893602    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" volumeName="kubernetes.io/projected/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-kube-api-access-cxjxd" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893609    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07cd964f-0f58-48bb-a2b5-81b1d0f355b2" volumeName="kubernetes.io/projected/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-kube-api-access-z2vks" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893617    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07cd964f-0f58-48bb-a2b5-81b1d0f355b2" volumeName="kubernetes.io/secret/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893627    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893640    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893650    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" volumeName="kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893657    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5703035-ada3-4cfb-b77b-4a8e9bcbe063" volumeName="kubernetes.io/projected/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-kube-api-access-2nmwq" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893664    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893672    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae4fdcf-846d-464a-a6a5-28713517c9aa" volumeName="kubernetes.io/configmap/3ae4fdcf-846d-464a-a6a5-28713517c9aa-config-volume" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893680    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893690    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893701    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2947ed62-ecb7-4ccd-935b-b1870cffb3e5" volumeName="kubernetes.io/projected/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-kube-api-access-wzxwt" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893710    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63acd3e4-e973-4124-afcf-39f855f0e4aa" volumeName="kubernetes.io/projected/63acd3e4-e973-4124-afcf-39f855f0e4aa-kube-api-access-7kv8n" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893736    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef120cf-f11d-435c-96c4-96d73f511afa" volumeName="kubernetes.io/projected/9ef120cf-f11d-435c-96c4-96d73f511afa-kube-api-access-tr87j" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893747    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893757    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16fc5076-624a-47e0-9430-9550882714d8" volumeName="kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893765    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef95306f-a39f-419e-84d7-6496c754701b" volumeName="kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893773    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7" volumeName="kubernetes.io/configmap/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-cabundle" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893781    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50141ec0-76c0-453c-9ecc-e5c7c227dad2" volumeName="kubernetes.io/secret/50141ec0-76c0-453c-9ecc-e5c7c227dad2-agent-certs" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893789    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="904ebe0f-c56d-4bd5-aae8-53dd560c8f5c" volumeName="kubernetes.io/projected/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-kube-api-access-khf9l" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893797    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9392d9f2-6a5c-4b6a-b048-a20227a79836" volumeName="kubernetes.io/projected/9392d9f2-6a5c-4b6a-b048-a20227a79836-kube-api-access-5qbjl" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893804    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bae3db47-36e8-4957-9c10-fd361815db36" volumeName="kubernetes.io/secret/bae3db47-36e8-4957-9c10-fd361815db36-serving-cert" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893814    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9807c6c6-ef8b-4769-bf2e-97b5c5a23604" volumeName="kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893827    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef95306f-a39f-419e-84d7-6496c754701b" volumeName="kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893837    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893856    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-certificates" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893864    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-trusted-ca" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893872    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893879    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce9cf1a-209e-489c-bae4-552a734e3e91" volumeName="kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-daemon-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893887    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94ef9fc0-8b2b-440a-bdd1-5684a5bc573d" volumeName="kubernetes.io/projected/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-kube-api-access-82gxq" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893896    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad0fe61-4d34-46c0-9b9f-263eb91dc103" volumeName="kubernetes.io/empty-dir/aad0fe61-4d34-46c0-9b9f-263eb91dc103-data-volume" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893908    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce9cf1a-209e-489c-bae4-552a734e3e91" volumeName="kubernetes.io/projected/1ce9cf1a-209e-489c-bae4-552a734e3e91-kube-api-access-tdcj2" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893920    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69cbe72d-a018-4a89-8951-a25e50d6d33b" volumeName="kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893931    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad0fe61-4d34-46c0-9b9f-263eb91dc103" volumeName="kubernetes.io/projected/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-api-access-mblft" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893944    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52" volumeName="kubernetes.io/projected/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-kube-api-access-jpjm7" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893955    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1a652a-2b2f-412a-bf99-3fab0319dd38" volumeName="kubernetes.io/projected/5a1a652a-2b2f-412a-bf99-3fab0319dd38-kube-api-access-k9dnr" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893963    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef120cf-f11d-435c-96c4-96d73f511afa" volumeName="kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-snapshots" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893971    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae4fdcf-846d-464a-a6a5-28713517c9aa" volumeName="kubernetes.io/projected/3ae4fdcf-846d-464a-a6a5-28713517c9aa-kube-api-access-9524c" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893983    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4357c52c-00a5-4bb2-bf8e-315d3760e129" volumeName="kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-tls" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.893994    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50141ec0-76c0-453c-9ecc-e5c7c227dad2" volumeName="kubernetes.io/configmap/50141ec0-76c0-453c-9ecc-e5c7c227dad2-konnectivity-ca" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894006    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1a652a-2b2f-412a-bf99-3fab0319dd38" volumeName="kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-tmp" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894016    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ef120cf-f11d-435c-96c4-96d73f511afa" volumeName="kubernetes.io/secret/9ef120cf-f11d-435c-96c4-96d73f511afa-serving-cert" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894022    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894033    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07cd964f-0f58-48bb-a2b5-81b1d0f355b2" volumeName="kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894045    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894055    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8110d47e-1f9c-480d-a1b6-d4a2b2b1e911" volumeName="kubernetes.io/projected/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911-kube-api-access-f8vm2" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894067    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894079    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894089    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e77045a-8555-4a49-9ccd-47a14931763b" volumeName="kubernetes.io/projected/7e77045a-8555-4a49-9ccd-47a14931763b-kube-api-access-xnx8r" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894100    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894113    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22f8456e-5749-4025-9161-4609e71ef8e8" volumeName="kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894126    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07cd964f-0f58-48bb-a2b5-81b1d0f355b2" volumeName="kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-config" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894135    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d852d5-b4c4-4f8f-b2da-bc84b039025e" volumeName="kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894166    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a559aa-cebd-40a6-9393-3feb287029ee" volumeName="kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894173    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" volumeName="kubernetes.io/projected/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-kube-api-access-8cbdf" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894180    6167 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc23c25a-4638-4c06-8804-3b1fe5135d56" volumeName="kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7" seLinuxMountContext=""
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894188    6167 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:45:44.894507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.894195    6167 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:45:44.905629 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.905607    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:45:44.908490 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.908469    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:45:44.918325 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.918282    6167 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:45:44.919683 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.919624    6167 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:45:44.919683 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.919654    6167 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:45:44.919825 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.919684    6167 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:45:44.919899 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:44.919846    6167 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 17:45:44.927795 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.927761    6167 generic.go:356] "Generic (PLEG): container finished" podID="f2eb497638654ab51d85f8c48e93d1a5" containerID="180c24f7be4aeea835a131d6a3fcc19e74743167c5ad5ad238e0ba517be31e8d" exitCode=0
Nov 03 17:45:44.952758 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.952733    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:45:44.952887 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.952771    6167 generic.go:356] "Generic (PLEG): container finished" podID="1f389b73-c992-49ac-ba2d-cf69b91f2b1e" containerID="24d401729f7b6dc365f11d8d9d49dd554a567be08e5d5cb26682037acea6a842" exitCode=255
Nov 03 17:45:44.989031 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:44.989010    6167 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:45:44.991916 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.991890    6167 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="2c72b5889ea8628b9894725d2ed18cb6c95f13cce8906cb26310566e88164368" exitCode=0
Nov 03 17:45:44.991916 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.991915    6167 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="96f21f245e5a3b2e7f932bd2d2f46fba1ebc0c5bb18f48b64ef5deac51fdb55a" exitCode=0
Nov 03 17:45:44.992059 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.991923    6167 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="88fe810013ea6547b61320f27dbf0c47dbec63da3fc58d3e425c7d44aa5b1e50" exitCode=0
Nov 03 17:45:44.992059 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.991929    6167 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="7d96b759f0ce1ab6ddec3bdf3894899774ccfa8b06314eadbf412bb00dfe190d" exitCode=0
Nov 03 17:45:44.992059 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.991934    6167 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="02fe4bab5770d0aea6c47b8829b991a82885a7d92aba722a359fc5b9b707a1b3" exitCode=0
Nov 03 17:45:44.992059 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:44.991939    6167 generic.go:356] "Generic (PLEG): container finished" podID="5bcba5ac-7a87-4aa9-b3e3-a474946ba957" containerID="06045d4c5f169da6d2b4933fbf2756d048d495713f2b6fda7eb56a668ab7bd69" exitCode=0
Nov 03 17:45:45.019932 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:45.019905    6167 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 17:45:45.029049 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.029031    6167 manager.go:324] Recovery completed
Nov 03 17:45:45.054572 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.054551    6167 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:45:45.056443 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.056421    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:45:45.056522 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.056455    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:45:45.056522 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.056466    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:45:45.059887 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.059865    6167 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:45:45.059887 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.059885    6167 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:45:45.060013 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.059886    6167 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:45:45.060013 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.059911    6167 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:45:45.060083 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060068    6167 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 17:45:45.060118 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060082    6167 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 17:45:45.060118 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060100    6167 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 17:45:45.060118 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060106    6167 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 17:45:45.060118 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060112    6167 policy_none.go:50] "Start"
Nov 03 17:45:45.060253 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060122    6167 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:45:45.060253 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060132    6167 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:45:45.060310 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060267    6167 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 17:45:45.060310 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060274    6167 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 17:45:45.060310 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060280    6167 policy_none.go:44] "Start"
Nov 03 17:45:45.060838 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060819    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:45:45.060910 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060858    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:45:45.060910 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.060874    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:45:45.065912 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.065881    6167 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:45:45.069611 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.069596    6167 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:45:45.069698 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:45.069626    6167 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:45:45.069698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.069638    6167 server.go:87] "Starting device plugin registration server"
Nov 03 17:45:45.069956 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.069942    6167 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:45:45.070020 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.069956    6167 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:45:45.070082 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.070045    6167 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:45:45.070234 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.070152    6167 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:45:45.070234 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.070193    6167 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:45:45.070234 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.070200    6167 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:45:45.070968 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.070268    6167 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:45:45.070183343Z","UUID":"9846592f-ac75-4ade-afc3-138f512f162e","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:45:45.072009 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.071992    6167 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:45:45.072081 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.072026    6167 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:45:45.072262 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:45.072244    6167 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:45:45.072334 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:45.072282    6167 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-108.us-west-2.compute.internal\" not found"
Nov 03 17:45:45.078761 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.078690    6167 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:45:45.079011 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.078980    6167 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 17:45:45.079133 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.079113    6167 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:45:45.129775 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.129748    6167 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:45:45.170819 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.170791    6167 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:45:45.173152 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.173133    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:45:45.173281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.173170    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:45:45.173281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.173182    6167 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-108.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:45:45.173281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.173231    6167 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.178370 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.178353    6167 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:45:45.220228 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.220184    6167 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"]
Nov 03 17:45:45.220929 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.220887    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"f2eb497638654ab51d85f8c48e93d1a5","Type":"ContainerStarted","Data":"0f82d9ec854958a3c0cecc68c256d1c2bc3db7a85ba717eacf0dd9a49f8dd939"}
Nov 03 17:45:45.221031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.220934    6167 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.221084 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.221074    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"acca05a04d25dcd6bb0ab5ae22e6769a","Type":"ContainerStarted","Data":"379bc65cd49430b284cbe11779f1a4409b30ee442272a39bca2cccf85b867595"}
Nov 03 17:45:45.221132 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.221090    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal" event={"ID":"acca05a04d25dcd6bb0ab5ae22e6769a","Type":"ContainerStarted","Data":"989f03362b49bc876376b5f2839fe316ea3bc0d76d0736fa732574f6880393ea"}
Nov 03 17:45:45.221132 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.221106    6167 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.284144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.284112    6167 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:45:45.284278 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:45.284162    6167 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.284993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.284974    6167 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:45:45.285083 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:45.285019    6167 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.286281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.286264    6167 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.286351 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.286328    6167 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.380883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.380810    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.380883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.380843    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acca05a04d25dcd6bb0ab5ae22e6769a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"acca05a04d25dcd6bb0ab5ae22e6769a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.380883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.380861    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.481299 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.481270    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.481434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.481316    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.481434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.481337    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acca05a04d25dcd6bb0ab5ae22e6769a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"acca05a04d25dcd6bb0ab5ae22e6769a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.481434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.481278    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.481434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.481374    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/acca05a04d25dcd6bb0ab5ae22e6769a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"acca05a04d25dcd6bb0ab5ae22e6769a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.481434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.481403    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2eb497638654ab51d85f8c48e93d1a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" (UID: \"f2eb497638654ab51d85f8c48e93d1a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:45.864675 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.864643    6167 apiserver.go:52] "Watching apiserver"
Nov 03 17:45:45.972966 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.972933    6167 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:45:45.973824 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:45.973792    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92","openshift-multus/multus-qkcgs","openshift-monitoring/node-exporter-6gk44","openshift-multus/network-metrics-daemon-8b682","openshift-console/downloads-6dd789c8b7-vfvbk","openshift-insights/insights-operator-7c4986d449-6nngz","openshift-image-registry/image-registry-69c466dc58-kmk8g","openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm","openshift-ingress-canary/ingress-canary-nkxg8","openshift-multus/multus-additional-cni-plugins-nmrf9","kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp","openshift-image-registry/image-registry-6dd4cffc4-zvgw7","openshift-ingress/router-default-7b45cc6f6f-ph9d6","openshift-insights/insights-runtime-extractor-hprg4","openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h","openshift-network-console/networking-console-plugin-86c478bb88-kkbq7","openshift-insights/periodic-gathering-4pbtv-w76t9","openshift-ovn-kubernetes/ovnkube-node-2crz8","openshift-console-operator/console-operator-7cd56b8dbc-w2mkr","openshift-dns/node-resolver-7mnmh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c","kube-system/global-pull-secret-syncer-zc2rm","openshift-image-registry/node-ca-8z24k","openshift-dns/dns-default-lrmc7","kube-system/kas-connection-checker-685895c7c9-65f6r","openshift-cluster-node-tuning-operator/tuned-m8nvv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4","openshift-console/console-6797c4cb4d-zx29b","openshift-console/console-574b5dd6dd-f4459","kube-system/kas-connection-checker-685895c7c9-txjqv","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l","openshift-service-ca/service-ca-5fffc97bdf-m5xh8","openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm","openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw","kube-system/kas-connection-checker-685895c7c9-7dsfq","openshift-network-diagnostics/network-check-target-jrt48","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c","openshift-network-operator/iptables-alerter-hn245","kube-system/konnectivity-agent-j2vd6"]
Nov 03 17:45:46.001684 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.001645    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.013632 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.013607    6167 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:46.013810 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.013792    6167 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:46.026426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.026404    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.027585 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.027550    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.027585 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.027570    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 17:45:46.027585 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.027551    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.028270 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.028246    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 17:45:46.028800 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.028782    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 17:45:46.028995 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.028972    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.028995 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.028977    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 17:45:46.029195 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.029048    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 17:45:46.029195 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.029058    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.029195 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.029100    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 17:45:46.029337 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.029218    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.029337 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.029237    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.030440    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.030817    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.030837    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.030911    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.030923    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.030957    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.031005    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.031048    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.031080    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.031089    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.031094    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 17:45:46.031626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.031222    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 17:45:46.032371 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.032350    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kf4nv\""
Nov 03 17:45:46.032581 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.032556    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 17:45:46.032671 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.032571    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 17:45:46.033000 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.032983    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 17:45:46.033122 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.033105    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 17:45:46.084470 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084443    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-system-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.084620 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084480    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vm2\" (UniqueName: \"kubernetes.io/projected/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911-kube-api-access-f8vm2\") pod \"kas-connection-checker-685895c7c9-7dsfq\" (UID: \"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911\") " pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:46.084620 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084499    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.084620 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084537    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-config\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:46.084620 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084597    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084648    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-bin\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084681    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084708    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bae3db47-36e8-4957-9c10-fd361815db36-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084759    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbdf\" (UniqueName: \"kubernetes.io/projected/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-kube-api-access-8cbdf\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084785    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084808    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:46.084842 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084833    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr87j\" (UniqueName: \"kubernetes.io/projected/9ef120cf-f11d-435c-96c4-96d73f511afa-kube-api-access-tr87j\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084873    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-netns\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084910    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084937    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-var-lib-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084962    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-netd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.084989    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-conf-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085013    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085028    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085038    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjm7\" (UniqueName: \"kubernetes.io/projected/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-kube-api-access-jpjm7\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085065    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-conf\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.085144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085090    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-hostroot\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085171    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmwq\" (UniqueName: \"kubernetes.io/projected/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-kube-api-access-2nmwq\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085207    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-serving-cert\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085227    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-config\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085233    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08a559aa-cebd-40a6-9393-3feb287029ee-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085260    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-socket-dir-parent\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085351    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-bin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085394    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxwt\" (UniqueName: \"kubernetes.io/projected/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-kube-api-access-wzxwt\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085426    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:46.085566 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085524    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-trusted-ca\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085581    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bae3db47-36e8-4957-9c10-fd361815db36-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085626    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-tmp\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085658    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bae3db47-36e8-4957-9c10-fd361815db36-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085697    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-host\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085765    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45jf\" (UniqueName: \"kubernetes.io/projected/16fc5076-624a-47e0-9430-9550882714d8-kube-api-access-p45jf\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085789    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-serving-cert\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085819    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovn-node-metrics-cert\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085863    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-daemon-config\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.085861    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-tmp\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086032    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-sys\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086041    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bae3db47-36e8-4957-9c10-fd361815db36-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:46.086057 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086050    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-netns\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086094    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086126    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086153    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbjl\" (UniqueName: \"kubernetes.io/projected/9392d9f2-6a5c-4b6a-b048-a20227a79836-kube-api-access-5qbjl\") pod \"kas-connection-checker-685895c7c9-65f6r\" (UID: \"9392d9f2-6a5c-4b6a-b048-a20227a79836\") " pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086185    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-etc-kubernetes\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086205    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gxq\" (UniqueName: \"kubernetes.io/projected/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-kube-api-access-82gxq\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086235    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086276    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086293    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086316    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-default-certificate\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086344    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-stats-auth\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086372    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-kubelet-config\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086399    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086427    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/2fd2bd81-c082-46e4-87c2-226f1acf6c07-kube-api-access-5th4k\") pod \"network-check-source-6f78d59f99-nwhnp\" (UID: \"2fd2bd81-c082-46e4-87c2-226f1acf6c07\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086450    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/16fc5076-624a-47e0-9430-9550882714d8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086451    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086493    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e77045a-8555-4a49-9ccd-47a14931763b-host-slash\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086534    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-dbus\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086565    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-default-certificate\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.086601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086568    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-var-lib-kubelet\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086675    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-ovn\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086711    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cnibin\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086758    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-stats-auth\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086772    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-cnibin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086820    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/69cbe72d-a018-4a89-8951-a25e50d6d33b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086864    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysconfig\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086937    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.086974    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087061    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-kubelet\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087093    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-systemd\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087120    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087141    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-k8s-cni-cncf-io\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087153    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-service-ca-bundle\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087181    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-metrics-certs\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087197    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087236    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-serving-cert\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087278    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2rc\" (UniqueName: \"kubernetes.io/projected/4e4803ee-0fee-44d8-a5d9-7931656d6e7c-kube-api-access-rk2rc\") pod \"volume-data-source-validator-58c9f5f46d-h8s5c\" (UID: \"4e4803ee-0fee-44d8-a5d9-7931656d6e7c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087391    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9984e6b-2191-4627-96ac-9d0b53762a46-hosts-file\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087413    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-lib-modules\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087441    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.087486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087482    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087504    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4jh\" (UniqueName: \"kubernetes.io/projected/b9984e6b-2191-4627-96ac-9d0b53762a46-kube-api-access-4c4jh\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087532    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-modprobe-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087576    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-serving-cert\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087615    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087638    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-kubelet\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087669    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-system-cni-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087699    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29nt9\" (UniqueName: \"kubernetes.io/projected/69cbe72d-a018-4a89-8951-a25e50d6d33b-kube-api-access-29nt9\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087755    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-host\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087858    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dnr\" (UniqueName: \"kubernetes.io/projected/5a1a652a-2b2f-412a-bf99-3fab0319dd38-kube-api-access-k9dnr\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087899    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-slash\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.087960    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcj2\" (UniqueName: \"kubernetes.io/projected/1ce9cf1a-209e-489c-bae4-552a734e3e91-kube-api-access-tdcj2\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088033    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-service-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088162    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088233    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-tuned\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088262    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-config\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088290    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-config\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.088327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088314    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-snapshots\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088344    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef120cf-f11d-435c-96c4-96d73f511afa-serving-cert\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088362    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-tuned\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088405    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088433    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-kubernetes\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088461    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-log-socket\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088488    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-env-overrides\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088513    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-cni-binary-copy\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088534    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-config\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088541    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088603    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-snapshots\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088627    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef120cf-f11d-435c-96c4-96d73f511afa-service-ca-bundle\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088639    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-systemd-units\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088700    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpctd\" (UniqueName: \"kubernetes.io/projected/bae3db47-36e8-4957-9c10-fd361815db36-kube-api-access-zpctd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088765    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-multus-certs\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088797    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-systemd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088831    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef120cf-f11d-435c-96c4-96d73f511afa-serving-cert\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088851    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vks\" (UniqueName: \"kubernetes.io/projected/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-kube-api-access-z2vks\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088871    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-run\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088891    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-etc-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088928    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-script-lib\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088958    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e77045a-8555-4a49-9ccd-47a14931763b-iptables-alerter-script\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.088984    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-serviceca\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089012    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7ds\" (UniqueName: \"kubernetes.io/projected/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5-kube-api-access-4m7ds\") pod \"kas-connection-checker-685895c7c9-txjqv\" (UID: \"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5\") " pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:46.089102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089037    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-node-log\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089115    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-multus\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089167    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089303    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-tmp\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089334    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089386    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjxd\" (UniqueName: \"kubernetes.io/projected/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-kube-api-access-cxjxd\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089408    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef120cf-f11d-435c-96c4-96d73f511afa-tmp\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089455    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089478    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089498    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089546    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9984e6b-2191-4627-96ac-9d0b53762a46-tmp-dir\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089589    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx8r\" (UniqueName: \"kubernetes.io/projected/7e77045a-8555-4a49-9ccd-47a14931763b-kube-api-access-xnx8r\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.089937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.089646    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9984e6b-2191-4627-96ac-9d0b53762a46-tmp-dir\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.090535 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.090519    6167 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:45:46.091487 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.091470    6167 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:45:46.091576 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:46.091512    6167 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:46.092110 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.092096    6167 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:45:46.092178 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:46.092131    6167 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-108.us-west-2.compute.internal"
Nov 03 17:45:46.127862 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.127788    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:45:46.127862 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.127836    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.127862 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.127836    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.128068 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.127906    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.128068 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.127964    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.128068 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.128000    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:45:46.128068 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.128022    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.128068 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.128031    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.130618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.130527    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.130618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.130549    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.130618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.130611    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:45:46.130828 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.130757    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.130828 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.130787    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.131365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131211    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.131365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131268    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:45:46.131486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131387    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.131486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131401    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 17:45:46.131486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131405    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.131486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131427    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 17:45:46.131486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131484    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.131653 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131641    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.131698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131652    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:45:46.131698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131662    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:45:46.131698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131644    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.131840 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131818    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:45:46.131886 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131838    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.132046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131985    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:45:46.132046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131996    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:45:46.132046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.131987    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.132046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132014    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:45:46.132046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132040    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 17:45:46.132046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132044    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:45:46.132272 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132064    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.132272 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132137    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.132272 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132209    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.132272 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132253    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:45:46.132423 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132294    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 17:45:46.132423 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132305    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.132423 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132316    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 17:45:46.132423 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132344    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:45:46.132423 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132418    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 17:45:46.132617 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132439    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:45:46.132617 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132477    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.132617 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132483    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 17:45:46.132617 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132479    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.132617 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132605    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 17:45:46.132816 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132529    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:45:46.132816 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132664    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.132816 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132738    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.133016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132854    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.133016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132911    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:45:46.133016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132918    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8lnxv\""
Nov 03 17:45:46.133016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.132939    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bmd8j\""
Nov 03 17:45:46.133016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133009    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z56rs\""
Nov 03 17:45:46.133016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133015    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133062    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133104    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133118    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7l2n\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133139    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xfh9g\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133012    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133209    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jd7fn\""
Nov 03 17:45:46.133311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133281    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133334    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133390    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mpkll\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133449    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6rhnc\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133461    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133484    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133517    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133529    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133539    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rfn9h\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133566    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133451    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\""
Nov 03 17:45:46.133616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133599    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133636    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133640    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-556wq\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133485    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xlmkc\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133670    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t67n\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133708    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7lgsk\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133759    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133908    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m6lsw\""
Nov 03 17:45:46.134031 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.133923    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bx8p5\""
Nov 03 17:45:46.134352 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.134334    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8w5gn\""
Nov 03 17:45:46.134480 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.134379    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:45:46.134480 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.134432    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135132    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5hg79\""
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135141    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135177    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135219    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135291    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6r97c\""
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135343    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5hxz7\""
Nov 03 17:45:46.135518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135473    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.135918 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135668    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:45:46.135918 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135670    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08a559aa-cebd-40a6-9393-3feb287029ee-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:46.135918 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.135879    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 17:45:46.136047 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136024    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/08a559aa-cebd-40a6-9393-3feb287029ee-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kkbq7\" (UID: \"08a559aa-cebd-40a6-9393-3feb287029ee\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kkbq7"
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136239    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-daemon-config\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136314    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovn-node-metrics-cert\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136323    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j4vc9\""
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136583    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136639    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gczt4\""
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.136879    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mk8dz\""
Nov 03 17:45:46.137111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.137062    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.137375 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.137157    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvbgk\""
Nov 03 17:45:46.138045 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.137777    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/69cbe72d-a018-4a89-8951-a25e50d6d33b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:46.138045 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.137906    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 17:45:46.138045 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.137980    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.138446 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.138429    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/69cbe72d-a018-4a89-8951-a25e50d6d33b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:46.138689 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.138663    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.139165    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce9cf1a-209e-489c-bae4-552a734e3e91-cni-binary-copy\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.139381    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-script-lib\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.139560    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-ovnkube-config\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.139713    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140042    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-original-pull-secret\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140134    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140242    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-metrics-certs\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140269    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140368    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-env-overrides\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140423    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e77045a-8555-4a49-9ccd-47a14931763b-iptables-alerter-script\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.141633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.140847    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-serviceca\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.144658 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.144636    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68wl\" (UniqueName: \"kubernetes.io/projected/8d645b31-5684-42a1-b305-59fe985149c4-kube-api-access-b68wl\") pod \"network-check-target-jrt48\" (UID: \"8d645b31-5684-42a1-b305-59fe985149c4\") " pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:46.144712 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.144667    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vm2\" (UniqueName: \"kubernetes.io/projected/8110d47e-1f9c-480d-a1b6-d4a2b2b1e911-kube-api-access-f8vm2\") pod \"kas-connection-checker-685895c7c9-7dsfq\" (UID: \"8110d47e-1f9c-480d-a1b6-d4a2b2b1e911\") " pod="kube-system/kas-connection-checker-685895c7c9-7dsfq"
Nov 03 17:45:46.145120 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.145104    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbdf\" (UniqueName: \"kubernetes.io/projected/1f389b73-c992-49ac-ba2d-cf69b91f2b1e-kube-api-access-8cbdf\") pod \"console-operator-7cd56b8dbc-w2mkr\" (UID: \"1f389b73-c992-49ac-ba2d-cf69b91f2b1e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:46.145854 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.145833    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmwq\" (UniqueName: \"kubernetes.io/projected/d5703035-ada3-4cfb-b77b-4a8e9bcbe063-kube-api-access-2nmwq\") pod \"network-metrics-daemon-8b682\" (UID: \"d5703035-ada3-4cfb-b77b-4a8e9bcbe063\") " pod="openshift-multus/network-metrics-daemon-8b682"
Nov 03 17:45:46.146834 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.146812    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.146904 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.146853    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjm7\" (UniqueName: \"kubernetes.io/projected/11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52-kube-api-access-jpjm7\") pod \"service-ca-operator-544d749c8c-twbtm\" (UID: \"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm"
Nov 03 17:45:46.147021 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.147008    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxwt\" (UniqueName: \"kubernetes.io/projected/2947ed62-ecb7-4ccd-935b-b1870cffb3e5-kube-api-access-wzxwt\") pod \"router-default-7b45cc6f6f-ph9d6\" (UID: \"2947ed62-ecb7-4ccd-935b-b1870cffb3e5\") " pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:46.147065 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.147010    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr87j\" (UniqueName: \"kubernetes.io/projected/9ef120cf-f11d-435c-96c4-96d73f511afa-kube-api-access-tr87j\") pod \"insights-operator-7c4986d449-6nngz\" (UID: \"9ef120cf-f11d-435c-96c4-96d73f511afa\") " pod="openshift-insights/insights-operator-7c4986d449-6nngz"
Nov 03 17:45:46.147633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.147613    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbjl\" (UniqueName: \"kubernetes.io/projected/9392d9f2-6a5c-4b6a-b048-a20227a79836-kube-api-access-5qbjl\") pod \"kas-connection-checker-685895c7c9-65f6r\" (UID: \"9392d9f2-6a5c-4b6a-b048-a20227a79836\") " pod="kube-system/kas-connection-checker-685895c7c9-65f6r"
Nov 03 17:45:46.149353 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.149336    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.149694 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.149676    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dnr\" (UniqueName: \"kubernetes.io/projected/5a1a652a-2b2f-412a-bf99-3fab0319dd38-kube-api-access-k9dnr\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.150191 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.150175    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29nt9\" (UniqueName: \"kubernetes.io/projected/69cbe72d-a018-4a89-8951-a25e50d6d33b-kube-api-access-29nt9\") pod \"cluster-monitoring-operator-884576c57-8ff8h\" (UID: \"69cbe72d-a018-4a89-8951-a25e50d6d33b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8ff8h"
Nov 03 17:45:46.150280 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.150265    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gxq\" (UniqueName: \"kubernetes.io/projected/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-kube-api-access-82gxq\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.151229 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.151214    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcj2\" (UniqueName: \"kubernetes.io/projected/1ce9cf1a-209e-489c-bae4-552a734e3e91-kube-api-access-tdcj2\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.151390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.151370    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/2fd2bd81-c082-46e4-87c2-226f1acf6c07-kube-api-access-5th4k\") pod \"network-check-source-6f78d59f99-nwhnp\" (UID: \"2fd2bd81-c082-46e4-87c2-226f1acf6c07\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwhnp"
Nov 03 17:45:46.151541 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.151527    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45jf\" (UniqueName: \"kubernetes.io/projected/16fc5076-624a-47e0-9430-9550882714d8-kube-api-access-p45jf\") pod \"cluster-samples-operator-6ddb6c8b94-rx64c\" (UID: \"16fc5076-624a-47e0-9430-9550882714d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rx64c"
Nov 03 17:45:46.152017 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.152002    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vks\" (UniqueName: \"kubernetes.io/projected/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-kube-api-access-z2vks\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.152324 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.152308    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2rc\" (UniqueName: \"kubernetes.io/projected/4e4803ee-0fee-44d8-a5d9-7931656d6e7c-kube-api-access-rk2rc\") pod \"volume-data-source-validator-58c9f5f46d-h8s5c\" (UID: \"4e4803ee-0fee-44d8-a5d9-7931656d6e7c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-h8s5c"
Nov 03 17:45:46.152495 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.152480    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjxd\" (UniqueName: \"kubernetes.io/projected/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-kube-api-access-cxjxd\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.152709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.152691    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") pod \"image-registry-6589fbf699-mdv92\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") " pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:45:46.152962 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.152947    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4jh\" (UniqueName: \"kubernetes.io/projected/b9984e6b-2191-4627-96ac-9d0b53762a46-kube-api-access-4c4jh\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.153531 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.153515    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpctd\" (UniqueName: \"kubernetes.io/projected/bae3db47-36e8-4957-9c10-fd361815db36-kube-api-access-zpctd\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-7mpds\" (UID: \"bae3db47-36e8-4957-9c10-fd361815db36\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds"
Nov 03 17:45:46.154899 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.154880    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx8r\" (UniqueName: \"kubernetes.io/projected/7e77045a-8555-4a49-9ccd-47a14931763b-kube-api-access-xnx8r\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.158428 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.158410    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7ds\" (UniqueName: \"kubernetes.io/projected/0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5-kube-api-access-4m7ds\") pod \"kas-connection-checker-685895c7c9-txjqv\" (UID: \"0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5\") " pod="kube-system/kas-connection-checker-685895c7c9-txjqv"
Nov 03 17:45:46.190468 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190446    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-systemd-units\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.190542 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190473    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.190542 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190490    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae4fdcf-846d-464a-a6a5-28713517c9aa-tmp-dir\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.190542 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190505    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-trusted-ca\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.190641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190568    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae4fdcf-846d-464a-a6a5-28713517c9aa-tmp-dir\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.190641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190567    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-systemd-units\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.190641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190615    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hxz\" (UniqueName: \"kubernetes.io/projected/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00-kube-api-access-p8hxz\") pod \"migrator-66df5d496b-k6qjm\" (UID: \"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:46.190641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190639    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:46.190846 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190659    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-multus-certs\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.190846 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190713    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-multus-certs\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.190846 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190739    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190834    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-systemd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190883    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-bound-sa-token\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190904    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190915    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-cabundle\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190928    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblft\" (UniqueName: \"kubernetes.io/projected/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-api-access-mblft\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190942    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-systemd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190965    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-run\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190981    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-trusted-ca\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.190993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.190994    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-etc-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191020    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191042    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-etc-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191047    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfpt\" (UniqueName: \"kubernetes.io/projected/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-kube-api-access-bxfpt\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191071    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50141ec0-76c0-453c-9ecc-e5c7c227dad2-konnectivity-ca\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191045    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-run\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191141    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191153    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad0fe61-4d34-46c0-9b9f-263eb91dc103-data-volume\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191195    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aad0fe61-4d34-46c0-9b9f-263eb91dc103-crio-socket\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191224    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-node-log\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191242    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50141ec0-76c0-453c-9ecc-e5c7c227dad2-konnectivity-ca\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191217    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad0fe61-4d34-46c0-9b9f-263eb91dc103-data-volume\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191246    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-wtmp\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191289    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khf9l\" (UniqueName: \"kubernetes.io/projected/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-kube-api-access-khf9l\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191288    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191312    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-node-log\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191348    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191451    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191490    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-multus\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.191509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191521    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-cert\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191543    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191559    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191582    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-multus\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191586    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191615    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191646    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191698    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191758    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-metrics-client-ca\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191763    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aad0fe61-4d34-46c0-9b9f-263eb91dc103-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191783    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191845    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmrv\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-kube-api-access-2fmrv\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191860    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-cert\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191875    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-system-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191887    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191902    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-bin\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191926    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae4fdcf-846d-464a-a6a5-28713517c9aa-config-volume\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191943    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-system-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191952    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191983    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-bin\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.191990    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192017    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-netns\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192039    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-root\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192040    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192075    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-image-registry-private-configuration\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192076    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-run-netns\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192098    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzbb\" (UniqueName: \"kubernetes.io/projected/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kube-api-access-jwzbb\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192104    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae4fdcf-846d-464a-a6a5-28713517c9aa-config-volume\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.192108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192114    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbv9j\" (UniqueName: \"kubernetes.io/projected/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-kube-api-access-jbv9j\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192240    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192252    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-image-registry-private-configuration\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192269    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-var-lib-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192295    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-netd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192320    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192332    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-var-lib-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192364    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-cni-netd\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192368    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-conf-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192402    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-conf\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192412    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192424    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192465    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-hostroot\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192466    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-conf-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192511    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-hostroot\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192522    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysctl-conf\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192529    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192560    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-socket-dir-parent\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192584    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-bin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192608    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192613    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-socket-dir-parent\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192634    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192641    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-cni-bin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192658    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-device-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192694    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kv8n\" (UniqueName: \"kubernetes.io/projected/63acd3e4-e973-4124-afcf-39f855f0e4aa-kube-api-access-7kv8n\") pod \"migrator-66df5d496b-cnjjw\" (UID: \"63acd3e4-e973-4124-afcf-39f855f0e4aa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192749    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192868    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9524c\" (UniqueName: \"kubernetes.io/projected/3ae4fdcf-846d-464a-a6a5-28713517c9aa-kube-api-access-9524c\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192898    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357c52c-00a5-4bb2-bf8e-315d3760e129-ca-trust-extracted\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192930    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-installation-pull-secrets\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192964    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-host\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.192997    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4357c52c-00a5-4bb2-bf8e-315d3760e129-ca-trust-extracted\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193007    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193032    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-sys\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193050    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94ef9fc0-8b2b-440a-bdd1-5684a5bc573d-host\") pod \"node-ca-8z24k\" (UID: \"94ef9fc0-8b2b-440a-bdd1-5684a5bc573d\") " pod="openshift-image-registry/node-ca-8z24k"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193056    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193099    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4357c52c-00a5-4bb2-bf8e-315d3760e129-installation-pull-secrets\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193103    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193104    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-sys\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193122    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-certificates\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193145    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-netns\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193164    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193180    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193195    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-key\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193210    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-etc-kubernetes\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193222    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-netns\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193251    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-etc-kubernetes\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193277    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193277    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193279    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193309    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-certificates\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.193281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193298    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193350    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-kubelet-config\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193378    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-signing-key\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193376    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193407    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193410    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-openvswitch\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193423    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50141ec0-76c0-453c-9ecc-e5c7c227dad2-agent-certs\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193426    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-kubelet-config\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193464    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193498    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193525    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193544    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193552    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193578    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e77045a-8555-4a49-9ccd-47a14931763b-host-slash\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193605    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-dbus\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193643    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9807c6c6-ef8b-4769-bf2e-97b5c5a23604-dbus\") pod \"global-pull-secret-syncer-zc2rm\" (UID: \"9807c6c6-ef8b-4769-bf2e-97b5c5a23604\") " pod="kube-system/global-pull-secret-syncer-zc2rm"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193660    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-var-lib-kubelet\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193668    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-multus-cni-dir\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193689    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-ovn\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193694    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e77045a-8555-4a49-9ccd-47a14931763b-host-slash\") pod \"iptables-alerter-hn245\" (UID: \"7e77045a-8555-4a49-9ccd-47a14931763b\") " pod="openshift-network-operator/iptables-alerter-hn245"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193702    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50141ec0-76c0-453c-9ecc-e5c7c227dad2-agent-certs\") pod \"konnectivity-agent-j2vd6\" (UID: \"50141ec0-76c0-453c-9ecc-e5c7c227dad2\") " pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193702    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-var-lib-kubelet\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193745    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cnibin\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193759    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-run-ovn\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193802    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-cnibin\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193827    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-cnibin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193865    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysconfig\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193910    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193927    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-cnibin\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193933    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-sysconfig\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193940    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdnc\" (UniqueName: \"kubernetes.io/projected/33b55ac5-4039-4da9-81f2-b64fadc22435-kube-api-access-dtdnc\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193971    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.193999    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194027    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-tmp\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194044    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-kubelet\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194058    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-systemd\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194091    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-textfile\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194110    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-var-lib-kubelet\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194115    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-tls\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194141    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194146    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-systemd\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194142    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-tmp\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194165    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-k8s-cni-cncf-io\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194209    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ce9cf1a-209e-489c-bae4-552a734e3e91-host-run-k8s-cni-cncf-io\") pod \"multus-qkcgs\" (UID: \"1ce9cf1a-209e-489c-bae4-552a734e3e91\") " pod="openshift-multus/multus-qkcgs"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194221    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ae4fdcf-846d-464a-a6a5-28713517c9aa-metrics-tls\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194225    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9984e6b-2191-4627-96ac-9d0b53762a46-hosts-file\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194266    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194274    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-lib-modules\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194270    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9984e6b-2191-4627-96ac-9d0b53762a46-hosts-file\") pod \"node-resolver-7mnmh\" (UID: \"b9984e6b-2191-4627-96ac-9d0b53762a46\") " pod="openshift-dns/node-resolver-7mnmh"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194300    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194325    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-tls\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194352    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-modprobe-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194377    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-lib-modules\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194378    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-kubelet\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194413    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-system-cni-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194443    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194412    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-kubelet\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194466    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194376    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194350    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194485    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194441    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-modprobe-d\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194517    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-host\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194447    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bcba5ac-7a87-4aa9-b3e3-a474946ba957-system-cni-dir\") pod \"multus-additional-cni-plugins-nmrf9\" (UID: \"5bcba5ac-7a87-4aa9-b3e3-a474946ba957\") " pod="openshift-multus/multus-additional-cni-plugins-nmrf9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194554    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-host\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194561    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-slash\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194589    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194591    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-host-slash\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194621    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194554    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-registry-tls\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194676    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194684    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194688    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hdr\" (UniqueName: \"kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194703    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194701    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194748    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/36506792-bfe4-4a9f-b27c-c2096cb67ebe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v9k7l\" (UID: \"36506792-bfe4-4a9f-b27c-c2096cb67ebe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194783    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-sys\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194810    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194836    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194902    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-kubernetes\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194921    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-log-socket\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194982    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07cd964f-0f58-48bb-a2b5-81b1d0f355b2-log-socket\") pod \"ovnkube-node-2crz8\" (UID: \"07cd964f-0f58-48bb-a2b5-81b1d0f355b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.194980    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/36506792-bfe4-4a9f-b27c-c2096cb67ebe-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-v9k7l\" (UID: \"36506792-bfe4-4a9f-b27c-c2096cb67ebe\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:46.195317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.195002    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a1a652a-2b2f-412a-bf99-3fab0319dd38-etc-kubernetes\") pod \"tuned-m8nvv\" (UID: \"5a1a652a-2b2f-412a-bf99-3fab0319dd38\") " pod="openshift-cluster-node-tuning-operator/tuned-m8nvv"
Nov 03 17:45:46.250483 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.250406    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hxz\" (UniqueName: \"kubernetes.io/projected/fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00-kube-api-access-p8hxz\") pod \"migrator-66df5d496b-k6qjm\" (UID: \"fc4eaf38-4ab2-4a04-91e5-bb2e9b184c00\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-k6qjm"
Nov 03 17:45:46.250774 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.250751    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-bound-sa-token\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.250880 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.250818    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblft\" (UniqueName: \"kubernetes.io/projected/aad0fe61-4d34-46c0-9b9f-263eb91dc103-kube-api-access-mblft\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.251135 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.251113    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khf9l\" (UniqueName: \"kubernetes.io/projected/904ebe0f-c56d-4bd5-aae8-53dd560c8f5c-kube-api-access-khf9l\") pod \"ingress-canary-nkxg8\" (UID: \"904ebe0f-c56d-4bd5-aae8-53dd560c8f5c\") " pod="openshift-ingress-canary/ingress-canary-nkxg8"
Nov 03 17:45:46.251301 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.251279    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzbb\" (UniqueName: \"kubernetes.io/projected/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kube-api-access-jwzbb\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.251651 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.251631    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbv9j\" (UniqueName: \"kubernetes.io/projected/2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7-kube-api-access-jbv9j\") pod \"service-ca-5fffc97bdf-m5xh8\" (UID: \"2b03107f-d2b0-4a5d-b404-9dd5bccb7dc7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-m5xh8"
Nov 03 17:45:46.251786 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.251752    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.252065 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.252047    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfpt\" (UniqueName: \"kubernetes.io/projected/4548c83e-c9e2-45fe-8ec0-3e450486cfd9-kube-api-access-bxfpt\") pod \"downloads-6dd789c8b7-vfvbk\" (UID: \"4548c83e-c9e2-45fe-8ec0-3e450486cfd9\") " pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:45:46.253662 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.253645    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") pod \"console-574b5dd6dd-f4459\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") " pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:46.256115 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.256090    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kv8n\" (UniqueName: \"kubernetes.io/projected/63acd3e4-e973-4124-afcf-39f855f0e4aa-kube-api-access-7kv8n\") pod \"migrator-66df5d496b-cnjjw\" (UID: \"63acd3e4-e973-4124-afcf-39f855f0e4aa\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cnjjw"
Nov 03 17:45:46.256217 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.256151    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") pod \"periodic-gathering-4pbtv-w76t9\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") " pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:45:46.256217 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.256214    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") pod \"image-registry-69c466dc58-kmk8g\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") " pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:45:46.256520 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.256499    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9524c\" (UniqueName: \"kubernetes.io/projected/3ae4fdcf-846d-464a-a6a5-28713517c9aa-kube-api-access-9524c\") pod \"dns-default-lrmc7\" (UID: \"3ae4fdcf-846d-464a-a6a5-28713517c9aa\") " pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:46.257187 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.257168    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmrv\" (UniqueName: \"kubernetes.io/projected/4357c52c-00a5-4bb2-bf8e-315d3760e129-kube-api-access-2fmrv\") pod \"image-registry-6dd4cffc4-zvgw7\" (UID: \"4357c52c-00a5-4bb2-bf8e-315d3760e129\") " pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:46.295308 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295275    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-wtmp\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295316    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295341    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295369    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-metrics-client-ca\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295397    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295399    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295439    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-root\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295451    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295453    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-wtmp\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.295464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295467    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295487    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-root\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295454    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295501    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295539    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295563    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-device-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295613    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295641    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295652    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-device-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295615    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295667    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295700    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295758    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295782    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdnc\" (UniqueName: \"kubernetes.io/projected/33b55ac5-4039-4da9-81f2-b64fadc22435-kube-api-access-dtdnc\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295790    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03b9dedb-2e8f-47d8-9a9f-3398edae5365-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwk4\" (UID: \"03b9dedb-2e8f-47d8-9a9f-3398edae5365\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwk4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295813    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-textfile\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295837    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-tls\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295876    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295899    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hdr\" (UniqueName: \"kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295926    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-sys\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295951    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.295975    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296023    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aad0fe61-4d34-46c0-9b9f-263eb91dc103-crio-socket\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296011    6167 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296096    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-metrics-client-ca\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296098    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aad0fe61-4d34-46c0-9b9f-263eb91dc103-crio-socket\") pod \"insights-runtime-extractor-hprg4\" (UID: \"aad0fe61-4d34-46c0-9b9f-263eb91dc103\") " pod="openshift-insights/insights-runtime-extractor-hprg4"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296170    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33b55ac5-4039-4da9-81f2-b64fadc22435-sys\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296200    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296272    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296514    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-textfile\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296552    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.297137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.296758    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-accelerators-collector-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.298615 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.297283    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.298615 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.297861    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.298750 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.298713    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-tls\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.298970 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.298951    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.299015 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.298968    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.299015 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.298985    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b55ac5-4039-4da9-81f2-b64fadc22435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.303279 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.303263    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:45:46.357310 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.357281    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdnc\" (UniqueName: \"kubernetes.io/projected/33b55ac5-4039-4da9-81f2-b64fadc22435-kube-api-access-dtdnc\") pod \"node-exporter-6gk44\" (UID: \"33b55ac5-4039-4da9-81f2-b64fadc22435\") " pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.357401 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.357313    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hdr\" (UniqueName: \"kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr\") pod \"console-6797c4cb4d-zx29b\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") " pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:46.602816 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.602784    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6gk44"
Nov 03 17:45:46.650248 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:46.650146    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b55ac5_4039_4da9_81f2_b64fadc22435.slice/crio-bbc4807de6137d5c2660afbaffc5f02dde39a8511cd8dafceb45b504ca02baab WatchSource:0}: Error finding container bbc4807de6137d5c2660afbaffc5f02dde39a8511cd8dafceb45b504ca02baab: Status 404 returned error can't find the container with id bbc4807de6137d5c2660afbaffc5f02dde39a8511cd8dafceb45b504ca02baab
Nov 03 17:45:46.654159 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:46.654075    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:47.023194 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:47.023112    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gk44" event={"ID":"33b55ac5-4039-4da9-81f2-b64fadc22435","Type":"ContainerStarted","Data":"bbc4807de6137d5c2660afbaffc5f02dde39a8511cd8dafceb45b504ca02baab"}
Nov 03 17:45:47.051220 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:47.051189    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab62d775_99b2_42a7_b44d_f24523284591.slice/crio-626712c04812cf8011857d7352e6465d28b771533e91f88b1b30888f3dbb9669 WatchSource:0}: Error finding container 626712c04812cf8011857d7352e6465d28b771533e91f88b1b30888f3dbb9669: Status 404 returned error can't find the container with id 626712c04812cf8011857d7352e6465d28b771533e91f88b1b30888f3dbb9669
Nov 03 17:45:47.051337 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:47.051229    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6797c4cb4d-zx29b"]
Nov 03 17:45:47.248430 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:47.248400    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c466dc58-kmk8g"]
Nov 03 17:45:48.030574 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.030531    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6797c4cb4d-zx29b" event={"ID":"ab62d775-99b2-42a7-b44d-f24523284591","Type":"ContainerStarted","Data":"be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962"}
Nov 03 17:45:48.031040 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.030629    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6797c4cb4d-zx29b" event={"ID":"ab62d775-99b2-42a7-b44d-f24523284591","Type":"ContainerStarted","Data":"626712c04812cf8011857d7352e6465d28b771533e91f88b1b30888f3dbb9669"}
Nov 03 17:45:48.202334 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.202303    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-w2mkr"
Nov 03 17:45:48.626657 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.626630    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:48.675819 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.675799    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:48.718438 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.718413    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:48.993295 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:48.993207    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6797c4cb4d-zx29b" podStartSLOduration=3.99319418 podStartE2EDuration="3.99319418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:48.992824793 +0000 UTC m=+4.223149865" watchObservedRunningTime="2026-11-03 17:45:48.99319418 +0000 UTC m=+4.223519252"
Nov 03 17:45:49.038218 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:49.038180    6167 generic.go:356] "Generic (PLEG): container finished" podID="33b55ac5-4039-4da9-81f2-b64fadc22435" containerID="86e202bb3cf5463f8ba4f9f493091183b7c7a1b45a3703e6e37922815bfbc617" exitCode=0
Nov 03 17:45:49.038653 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:49.038241    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gk44" event={"ID":"33b55ac5-4039-4da9-81f2-b64fadc22435","Type":"ContainerDied","Data":"86e202bb3cf5463f8ba4f9f493091183b7c7a1b45a3703e6e37922815bfbc617"}
Nov 03 17:45:49.038653 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:49.038502    6167 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:45:49.319675 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:49.319595    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:49.323265 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:49.323242    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-v9k7l"
Nov 03 17:45:50.047085 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.046515    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gk44" event={"ID":"33b55ac5-4039-4da9-81f2-b64fadc22435","Type":"ContainerStarted","Data":"5b4f36d725638d22419b4fc72eee41130bceee7fc5bd50eb11f26dc5cda54766"}
Nov 03 17:45:50.047085 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.046587    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6gk44" event={"ID":"33b55ac5-4039-4da9-81f2-b64fadc22435","Type":"ContainerStarted","Data":"b52845ecb5002e16ce6239430ac10f89d59cf5f9321a17097584675ff2b621e1"}
Nov 03 17:45:50.533170 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.533133    6167 patch_prober.go:29] interesting pod/image-registry-6dd4cffc4-zvgw7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:50.533317 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.533204    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7" podUID="4357c52c-00a5-4bb2-bf8e-315d3760e129" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:50.534277 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.534248    6167 patch_prober.go:29] interesting pod/image-registry-69c466dc58-kmk8g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:50.534410 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.534296    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:50.538670 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.538645    6167 patch_prober.go:29] interesting pod/image-registry-69c466dc58-kmk8g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:50.538802 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:50.538694    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:51.052111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.052078    6167 generic.go:356] "Generic (PLEG): container finished" podID="bae3db47-36e8-4957-9c10-fd361815db36" containerID="4d35fe6754dc2bb67af5ba9e41e5d73fa29dd392c62db10c6cf50cc89a312e7e" exitCode=0
Nov 03 17:45:51.052521 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.052170    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds" event={"ID":"bae3db47-36e8-4957-9c10-fd361815db36","Type":"ContainerDied","Data":"4d35fe6754dc2bb67af5ba9e41e5d73fa29dd392c62db10c6cf50cc89a312e7e"}
Nov 03 17:45:51.052608 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.052589    6167 scope.go:122] "RemoveContainer" containerID="4d35fe6754dc2bb67af5ba9e41e5d73fa29dd392c62db10c6cf50cc89a312e7e"
Nov 03 17:45:51.054016 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.053993    6167 generic.go:356] "Generic (PLEG): container finished" podID="9ef120cf-f11d-435c-96c4-96d73f511afa" containerID="0156fd83b62140dc89698ce0d83ebf758abdf90acd5d1720c815f83bc2eeaaba" exitCode=0
Nov 03 17:45:51.054101 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.054022    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6nngz" event={"ID":"9ef120cf-f11d-435c-96c4-96d73f511afa","Type":"ContainerDied","Data":"0156fd83b62140dc89698ce0d83ebf758abdf90acd5d1720c815f83bc2eeaaba"}
Nov 03 17:45:51.054359 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.054342    6167 scope.go:122] "RemoveContainer" containerID="0156fd83b62140dc89698ce0d83ebf758abdf90acd5d1720c815f83bc2eeaaba"
Nov 03 17:45:51.056485 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.056461    6167 generic.go:356] "Generic (PLEG): container finished" podID="11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52" containerID="8df8ab03b27a871a8bd2dc934fae9f42b35dd273020a91039cf8f2b3909fb98a" exitCode=0
Nov 03 17:45:51.056580 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.056552    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm" event={"ID":"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52","Type":"ContainerDied","Data":"8df8ab03b27a871a8bd2dc934fae9f42b35dd273020a91039cf8f2b3909fb98a"}
Nov 03 17:45:51.057081 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.057067    6167 scope.go:122] "RemoveContainer" containerID="8df8ab03b27a871a8bd2dc934fae9f42b35dd273020a91039cf8f2b3909fb98a"
Nov 03 17:45:51.631382 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:51.631352    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lrmc7"
Nov 03 17:45:52.062426 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:52.062334    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-twbtm" event={"ID":"11e00a5d-ac2d-4dc5-90cf-106cf9a4bf52","Type":"ContainerStarted","Data":"89ca8b47a4b09af610279c372526e69460645a36d8a39ed9ee427fb07bf5de53"}
Nov 03 17:45:52.064295 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:52.064265    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-7mpds" event={"ID":"bae3db47-36e8-4957-9c10-fd361815db36","Type":"ContainerStarted","Data":"548356857cdc87590551ff1cfa777dcbf5baf8c49de7831819681ab87273c77c"}
Nov 03 17:45:52.066040 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:52.066020    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6nngz" event={"ID":"9ef120cf-f11d-435c-96c4-96d73f511afa","Type":"ContainerStarted","Data":"07e1dfbd0e659f05f8c673749a9c890adf9b4db3351c4862b1baf6311ac52870"}
Nov 03 17:45:53.241574 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.241479    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:53.247215 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.247185    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:53.416022 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.415996    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2crz8"
Nov 03 17:45:53.742570 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.742523    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6797c4cb4d-zx29b"]
Nov 03 17:45:53.777795 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.777760    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6db7d59f8-xrzpx"]
Nov 03 17:45:53.780728 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.780703    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.838959 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.838928    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db7d59f8-xrzpx"]
Nov 03 17:45:53.870856 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.870825    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871032 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.870871    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871032 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.870924    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871032 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.870987    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871206 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.871066    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmv2\" (UniqueName: \"kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871206 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.871172    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871489 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.871215    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.871489 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.871242    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.888799 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.888767    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:53.972654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.972616    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.972654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.972659    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.972886 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.972680    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.972886 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.972744    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.973461 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.973432    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmv2\" (UniqueName: \"kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.973604 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.973475    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.973604 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.973520    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.973604 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.973549    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.973756 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.973643    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.974008 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.973980    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.974611 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.974581    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.974897 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.974876    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.975014 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.974944    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.975939 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.975918    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:53.977092 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:53.977064    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:54.036729 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.036612    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmv2\" (UniqueName: \"kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2\") pod \"console-6db7d59f8-xrzpx\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") " pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:54.093315 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.093269    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:54.364929 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.364891    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:45:54.471238 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.471164    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db7d59f8-xrzpx"]
Nov 03 17:45:54.472161 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:45:54.471844    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b62eb72_4187_4eb5_bb57_e48fdc9d5b98.slice/crio-029ec813e2355e0c58d6b795f1cef5380a6bdbf2e26ab326f9649d7d48cffd0c WatchSource:0}: Error finding container 029ec813e2355e0c58d6b795f1cef5380a6bdbf2e26ab326f9649d7d48cffd0c: Status 404 returned error can't find the container with id 029ec813e2355e0c58d6b795f1cef5380a6bdbf2e26ab326f9649d7d48cffd0c
Nov 03 17:45:54.527785 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:45:54.527747    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:45:54.581468 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.580823    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b45cc6f6f-ph9d6"
Nov 03 17:45:54.604783 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.604749    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:45:54.618327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.618252    6167 patch_prober.go:29] interesting pod/image-registry-6dd4cffc4-zvgw7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:54.618327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.618307    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7" podUID="4357c52c-00a5-4bb2-bf8e-315d3760e129" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:54.623436 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.623412    6167 patch_prober.go:29] interesting pod/image-registry-6dd4cffc4-zvgw7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 17:45:54.623565 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:54.623455    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7" podUID="4357c52c-00a5-4bb2-bf8e-315d3760e129" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:45:55.082742 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:55.082685    6167 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:45:55.082742 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:55.082689    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db7d59f8-xrzpx" event={"ID":"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98","Type":"ContainerStarted","Data":"d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267"}
Nov 03 17:45:55.082951 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:55.082780    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db7d59f8-xrzpx" event={"ID":"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98","Type":"ContainerStarted","Data":"029ec813e2355e0c58d6b795f1cef5380a6bdbf2e26ab326f9649d7d48cffd0c"}
Nov 03 17:45:55.202125 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:55.202074    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6db7d59f8-xrzpx" podStartSLOduration=2.202061262 podStartE2EDuration="2.202061262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:55.201533706 +0000 UTC m=+10.431858802" watchObservedRunningTime="2026-11-03 17:45:55.202061262 +0000 UTC m=+10.432386394"
Nov 03 17:45:55.907897 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:55.907865    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:55.911537 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:55.911517    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:45:56.083562 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:56.083534    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jrt48"
Nov 03 17:45:56.892779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:56.892746    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:45:59.539481 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.539449    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574b5dd6dd-f4459"]
Nov 03 17:45:59.589869 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.589840    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79b9d75875-9ztzl"]
Nov 03 17:45:59.593562 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.593537    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.648870 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.648807    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b9d75875-9ztzl"]
Nov 03 17:45:59.725810 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.725779    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76pg\" (UniqueName: \"kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726001 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.725831    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726001 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.725861    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726001 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.725979    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726167 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.726018    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726167 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.726049    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726167 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.726135    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.726325 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.726204    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.826995 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.826965    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827149 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827024    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827149 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827094    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827149 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827130    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76pg\" (UniqueName: \"kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827167    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827188    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827218    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827251    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827597 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827572    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.827942 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.827918    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.828135 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.828114    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.828272 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.828249    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.828358 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.828336    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.829630 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.829607    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.829738 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.829675    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.889125 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.889093    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76pg\" (UniqueName: \"kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg\") pod \"console-79b9d75875-9ztzl\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") " pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:45:59.907381 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:45:59.907356    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:00.296569 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:00.296535    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b9d75875-9ztzl"]
Nov 03 17:46:00.296786 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:46:00.296610    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad54663_9592_4ef3_865e_d92b39fc946b.slice/crio-72df771456ec7f497ae82f418ec0c6f96912f11aff8d8cda491cd0c3aefec3b6 WatchSource:0}: Error finding container 72df771456ec7f497ae82f418ec0c6f96912f11aff8d8cda491cd0c3aefec3b6: Status 404 returned error can't find the container with id 72df771456ec7f497ae82f418ec0c6f96912f11aff8d8cda491cd0c3aefec3b6
Nov 03 17:46:00.538833 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:00.538803    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:46:01.112788 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:01.112751    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9d75875-9ztzl" event={"ID":"dad54663-9592-4ef3-865e-d92b39fc946b","Type":"ContainerStarted","Data":"ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564"}
Nov 03 17:46:01.113239 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:01.112829    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9d75875-9ztzl" event={"ID":"dad54663-9592-4ef3-865e-d92b39fc946b","Type":"ContainerStarted","Data":"72df771456ec7f497ae82f418ec0c6f96912f11aff8d8cda491cd0c3aefec3b6"}
Nov 03 17:46:01.232902 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:01.232844    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79b9d75875-9ztzl" podStartSLOduration=2.232829292 podStartE2EDuration="2.232829292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:01.232019794 +0000 UTC m=+16.462344865" watchObservedRunningTime="2026-11-03 17:46:01.232829292 +0000 UTC m=+16.463154361"
Nov 03 17:46:04.623982 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:04.623946    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd4cffc4-zvgw7"
Nov 03 17:46:04.880023 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:04.879950    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:46:04.886405 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:04.886377    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:46:05.137850 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:05.137766    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:46:05.340952 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:05.340923    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j2vd6"
Nov 03 17:46:06.581879 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.581847    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4pbtv-w76t9"]
Nov 03 17:46:06.582308 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.582102    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4pbtv-w76t9" podUID="ef95306f-a39f-419e-84d7-6496c754701b" containerName="insights-gathering" containerID="cri-o://3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd" gracePeriod=30
Nov 03 17:46:06.843111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.843080    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:46:06.997999 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.997961    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") pod \"ef95306f-a39f-419e-84d7-6496c754701b\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") "
Nov 03 17:46:06.998161 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.998030    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") pod \"ef95306f-a39f-419e-84d7-6496c754701b\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") "
Nov 03 17:46:06.998161 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.998073    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") pod \"ef95306f-a39f-419e-84d7-6496c754701b\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") "
Nov 03 17:46:06.998161 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.998103    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") pod \"ef95306f-a39f-419e-84d7-6496c754701b\" (UID: \"ef95306f-a39f-419e-84d7-6496c754701b\") "
Nov 03 17:46:06.998863 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.998803    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle" pod "ef95306f-a39f-419e-84d7-6496c754701b" (UID: "ef95306f-a39f-419e-84d7-6496c754701b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:06.998970 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.998872    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path" pod "ef95306f-a39f-419e-84d7-6496c754701b" (UID: "ef95306f-a39f-419e-84d7-6496c754701b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:06.998970 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:06.998883    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle" pod "ef95306f-a39f-419e-84d7-6496c754701b" (UID: "ef95306f-a39f-419e-84d7-6496c754701b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:07.001174 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.001147    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz" pod "ef95306f-a39f-419e-84d7-6496c754701b" (UID: "ef95306f-a39f-419e-84d7-6496c754701b"). InnerVolumeSpecName "kube-api-access-k6xrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:07.099909 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.099820    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-trusted-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:07.099909 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.099855    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6xrz\" (UniqueName: \"kubernetes.io/projected/ef95306f-a39f-419e-84d7-6496c754701b-kube-api-access-k6xrz\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:07.099909 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.099872    6167 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef95306f-a39f-419e-84d7-6496c754701b-service-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:07.103506 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.103467    6167 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ef95306f-a39f-419e-84d7-6496c754701b-archives-path\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:07.141094 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.141054    6167 generic.go:356] "Generic (PLEG): container finished" podID="ef95306f-a39f-419e-84d7-6496c754701b" containerID="3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd" exitCode=2
Nov 03 17:46:07.141256 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.141102    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4pbtv-w76t9"
Nov 03 17:46:07.141256 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.141143    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4pbtv-w76t9" event={"ID":"ef95306f-a39f-419e-84d7-6496c754701b","Type":"ContainerDied","Data":"3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd"}
Nov 03 17:46:07.141256 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.141207    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4pbtv-w76t9" event={"ID":"ef95306f-a39f-419e-84d7-6496c754701b","Type":"ContainerDied","Data":"fa2a748e2bdb6ea8c3a60d8b624d7b639e4260a0d4962ab4e3d66cfe4991e7b9"}
Nov 03 17:46:07.141256 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.141236    6167 scope.go:122] "RemoveContainer" containerID="3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd"
Nov 03 17:46:07.152207 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.152186    6167 scope.go:122] "RemoveContainer" containerID="3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd"
Nov 03 17:46:07.152561 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:07.152534    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd\": container with ID starting with 3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd not found: ID does not exist" containerID="3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd"
Nov 03 17:46:07.152655 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.152577    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd"} err="failed to get container status \"3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd\": rpc error: code = NotFound desc = could not find container \"3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd\": container with ID starting with 3585ada03820fdbbd8752c0fd67b9bc5515ec97c38a197147116ca36dc65a7fd not found: ID does not exist"
Nov 03 17:46:07.322214 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.322178    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4pbtv-w76t9"]
Nov 03 17:46:07.325708 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:07.325684    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4pbtv-w76t9"]
Nov 03 17:46:08.927647 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:08.927610    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef95306f-a39f-419e-84d7-6496c754701b" path="/var/lib/kubelet/pods/ef95306f-a39f-419e-84d7-6496c754701b/volumes"
Nov 03 17:46:09.907968 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:09.907918    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:09.908160 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:09.908004    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:09.913189 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:09.913160    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:10.162202 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:10.162132    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:10.313990 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:10.313949    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db7d59f8-xrzpx"]
Nov 03 17:46:11.003647 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.003598    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerName="registry" containerID="cri-o://9ac86bccc6e897224c7b089768857317d7fa72c929475687f4ee97d5dce6f597" gracePeriod=30
Nov 03 17:46:11.164268 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.164239    6167 generic.go:356] "Generic (PLEG): container finished" podID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerID="9ac86bccc6e897224c7b089768857317d7fa72c929475687f4ee97d5dce6f597" exitCode=0
Nov 03 17:46:11.164659 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.164310    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" event={"ID":"18d852d5-b4c4-4f8f-b2da-bc84b039025e","Type":"ContainerDied","Data":"9ac86bccc6e897224c7b089768857317d7fa72c929475687f4ee97d5dce6f597"}
Nov 03 17:46:11.166033 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.166008    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vfvbk" event={"ID":"4548c83e-c9e2-45fe-8ec0-3e450486cfd9","Type":"ContainerStarted","Data":"52d2c78ca6b4fcc775b48f16150ef1b3f9f0b3fad8a6df05648dddf435265b42"}
Nov 03 17:46:11.166594 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.166378    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:46:11.167952 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.167927    6167 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-vfvbk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.28:8080/\": dial tcp 10.132.0.28:8080: connect: connection refused" start-of-body=
Nov 03 17:46:11.168045 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.167979    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-vfvbk" podUID="4548c83e-c9e2-45fe-8ec0-3e450486cfd9" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.28:8080/\": dial tcp 10.132.0.28:8080: connect: connection refused"
Nov 03 17:46:11.265273 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.265247    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:46:11.343139 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343111    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343364 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343160    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343364 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343227    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343364 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343257    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343364 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343328    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343364 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343366    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343406    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343451    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") pod \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\" (UID: \"18d852d5-b4c4-4f8f-b2da-bc84b039025e\") "
Nov 03 17:46:11.343649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343618    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:11.343824 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.343786    6167 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-certificates\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.344659 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.344629    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:11.347002 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.346962    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:11.347434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.347398    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "kube-api-access-79mh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:11.349897 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.349869    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:11.351350 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.351325    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:11.353090 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.353067    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:11.354791 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.354755    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token" pod "18d852d5-b4c4-4f8f-b2da-bc84b039025e" (UID: "18d852d5-b4c4-4f8f-b2da-bc84b039025e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:11.444924 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444887    6167 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-bound-sa-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.444924 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444918    6167 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18d852d5-b4c4-4f8f-b2da-bc84b039025e-ca-trust-extracted\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.444924 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444928    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18d852d5-b4c4-4f8f-b2da-bc84b039025e-trusted-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.445213 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444937    6167 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-installation-pull-secrets\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.445213 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444949    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79mh5\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-kube-api-access-79mh5\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.445213 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444962    6167 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18d852d5-b4c4-4f8f-b2da-bc84b039025e-registry-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:11.445213 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:11.444979    6167 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18d852d5-b4c4-4f8f-b2da-bc84b039025e-image-registry-private-configuration\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.173127 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.172398    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6589fbf699-mdv92" event={"ID":"18d852d5-b4c4-4f8f-b2da-bc84b039025e","Type":"ContainerDied","Data":"1ce03a1b15a00f973da3386bd6a86057a366acc347f812c46c7a8034f7169858"}
Nov 03 17:46:12.173127 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.172469    6167 scope.go:122] "RemoveContainer" containerID="9ac86bccc6e897224c7b089768857317d7fa72c929475687f4ee97d5dce6f597"
Nov 03 17:46:12.173127 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.172409    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6589fbf699-mdv92"
Nov 03 17:46:12.176819 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.176797    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vfvbk"
Nov 03 17:46:12.268598 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.268516    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry" containerID="cri-o://3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17" gracePeriod=30
Nov 03 17:46:12.513285 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.513247    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92"]
Nov 03 17:46:12.513443 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.513305    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6589fbf699-mdv92"]
Nov 03 17:46:12.609526 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.609490    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:46:12.758473 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758437    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758490    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758513    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758568    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758633    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758913 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758675    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758913 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758714    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.758913 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.758777    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") pod \"22f8456e-5749-4025-9161-4609e71ef8e8\" (UID: \"22f8456e-5749-4025-9161-4609e71ef8e8\") "
Nov 03 17:46:12.763927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.759490    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:12.763927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.763223    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:12.763927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.763884    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:12.766644 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.766600    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "kube-api-access-w9h75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:12.768809 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.768777    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:12.774337 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.772937    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:12.774337 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.774247    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:12.774476 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.774405    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration" pod "22f8456e-5749-4025-9161-4609e71ef8e8" (UID: "22f8456e-5749-4025-9161-4609e71ef8e8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859566    6167 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-image-registry-private-configuration\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859604    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9h75\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-kube-api-access-w9h75\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859618    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-trusted-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859627    6167 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22f8456e-5749-4025-9161-4609e71ef8e8-registry-certificates\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859637    6167 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-registry-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859646    6167 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22f8456e-5749-4025-9161-4609e71ef8e8-bound-sa-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.859648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859654    6167 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22f8456e-5749-4025-9161-4609e71ef8e8-ca-trust-extracted\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.860083 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.859663    6167 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22f8456e-5749-4025-9161-4609e71ef8e8-installation-pull-secrets\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:12.927928 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:12.927889    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" path="/var/lib/kubelet/pods/18d852d5-b4c4-4f8f-b2da-bc84b039025e/volumes"
Nov 03 17:46:13.180128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.180050    6167 generic.go:356] "Generic (PLEG): container finished" podID="22f8456e-5749-4025-9161-4609e71ef8e8" containerID="3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17" exitCode=0
Nov 03 17:46:13.180128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.180117    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g"
Nov 03 17:46:13.180641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.180137    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" event={"ID":"22f8456e-5749-4025-9161-4609e71ef8e8","Type":"ContainerDied","Data":"3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17"}
Nov 03 17:46:13.180641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.180189    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c466dc58-kmk8g" event={"ID":"22f8456e-5749-4025-9161-4609e71ef8e8","Type":"ContainerDied","Data":"58d215ead268604661374cc6f4e586c714af6d3dec0944028f363b59fea1a16d"}
Nov 03 17:46:13.180641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.180218    6167 scope.go:122] "RemoveContainer" containerID="3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17"
Nov 03 17:46:13.191125 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.191104    6167 scope.go:122] "RemoveContainer" containerID="3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17"
Nov 03 17:46:13.191452 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:13.191428    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17\": container with ID starting with 3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17 not found: ID does not exist" containerID="3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17"
Nov 03 17:46:13.191549 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.191467    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17"} err="failed to get container status \"3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17\": rpc error: code = NotFound desc = could not find container \"3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17\": container with ID starting with 3e60a90e7656b320c04ae5cc245a80857ca4d56d6ce9069cef7604a50ed4be17 not found: ID does not exist"
Nov 03 17:46:13.347783 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.347750    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c466dc58-kmk8g"]
Nov 03 17:46:13.353461 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:13.353434    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69c466dc58-kmk8g"]
Nov 03 17:46:14.927128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:14.927092    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" path="/var/lib/kubelet/pods/22f8456e-5749-4025-9161-4609e71ef8e8/volumes"
Nov 03 17:46:20.106291 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.106225    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6797c4cb4d-zx29b" podUID="ab62d775-99b2-42a7-b44d-f24523284591" containerName="console" containerID="cri-o://be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962" gracePeriod=15
Nov 03 17:46:20.394042 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.394014    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:46:20.534620 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534589    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.534792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534628    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.534792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534661    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.534792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534709    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.534792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534762    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.535193 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534831    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.535193 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534867    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr\" (UniqueName: \"kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.535193 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534915    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca\") pod \"ab62d775-99b2-42a7-b44d-f24523284591\" (UID: \"ab62d775-99b2-42a7-b44d-f24523284591\") "
Nov 03 17:46:20.535193 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.534957    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:20.535193 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.535037    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:20.535471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.535312    6167 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-console-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.535471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.535335    6167 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab62d775-99b2-42a7-b44d-f24523284591-tmp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.535471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.535381    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:20.535471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.535385    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:20.535471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.535420    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:20.537197 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.537169    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:20.537299 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.537203    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "kube-api-access-p8hdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:20.537464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.537448    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert" pod "ab62d775-99b2-42a7-b44d-f24523284591" (UID: "ab62d775-99b2-42a7-b44d-f24523284591"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:20.636367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.636292    6167 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-oauth-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.636367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.636322    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hdr\" (UniqueName: \"kubernetes.io/projected/ab62d775-99b2-42a7-b44d-f24523284591-kube-api-access-p8hdr\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.636367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.636335    6167 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.636367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.636344    6167 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-oauth-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.636367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.636353    6167 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab62d775-99b2-42a7-b44d-f24523284591-console-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.636367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.636361    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab62d775-99b2-42a7-b44d-f24523284591-trusted-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:20.873333 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873300    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d855b574f-m2t5m"]
Nov 03 17:46:20.873676 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873663    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ef95306f-a39f-419e-84d7-6496c754701b" containerName="insights-gathering"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873677    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef95306f-a39f-419e-84d7-6496c754701b" containerName="insights-gathering"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873687    6167 cpu_manager.go:414] "containerMap: removing container" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873692    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873705    6167 cpu_manager.go:414] "containerMap: removing container" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerName="registry"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873710    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerName="registry"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873745    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ab62d775-99b2-42a7-b44d-f24523284591" containerName="console"
Nov 03 17:46:20.873779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873753    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab62d775-99b2-42a7-b44d-f24523284591" containerName="console"
Nov 03 17:46:20.874009 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873807    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab62d775-99b2-42a7-b44d-f24523284591" containerName="console"
Nov 03 17:46:20.874009 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873821    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef95306f-a39f-419e-84d7-6496c754701b" containerName="insights-gathering"
Nov 03 17:46:20.874009 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873829    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f8456e-5749-4025-9161-4609e71ef8e8" containerName="registry"
Nov 03 17:46:20.874009 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.873838    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d852d5-b4c4-4f8f-b2da-bc84b039025e" containerName="registry"
Nov 03 17:46:20.906231 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.906178    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:20.975614 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:20.975585    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d855b574f-m2t5m"]
Nov 03 17:46:21.039497 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039462    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-service-ca\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039497 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039500    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmq4\" (UniqueName: \"kubernetes.io/projected/166e94e3-43f7-42e2-9f02-0758789334ed-kube-api-access-fkmq4\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039711 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039532    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-trusted-ca-bundle\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039711 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039548    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-oauth-serving-cert\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039711 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039703    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-console-config\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039832 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039766    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166e94e3-43f7-42e2-9f02-0758789334ed-console-serving-cert\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039832 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039783    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166e94e3-43f7-42e2-9f02-0758789334ed-console-oauth-config\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.039832 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.039801    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166e94e3-43f7-42e2-9f02-0758789334ed-tmp\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.140931 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.140896    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-service-ca\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.140931 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.140935    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmq4\" (UniqueName: \"kubernetes.io/projected/166e94e3-43f7-42e2-9f02-0758789334ed-kube-api-access-fkmq4\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.140965    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-trusted-ca-bundle\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.140983    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-oauth-serving-cert\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141008    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-console-config\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141025    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166e94e3-43f7-42e2-9f02-0758789334ed-console-serving-cert\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141039    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166e94e3-43f7-42e2-9f02-0758789334ed-console-oauth-config\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141055    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166e94e3-43f7-42e2-9f02-0758789334ed-tmp\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141789 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141440    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/166e94e3-43f7-42e2-9f02-0758789334ed-tmp\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141789 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141752    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-service-ca\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141843    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-oauth-serving-cert\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.141955 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.141915    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-console-config\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.142039 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.142021    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/166e94e3-43f7-42e2-9f02-0758789334ed-trusted-ca-bundle\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.143603 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.143577    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166e94e3-43f7-42e2-9f02-0758789334ed-console-serving-cert\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.143696 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.143668    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166e94e3-43f7-42e2-9f02-0758789334ed-console-oauth-config\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.203027 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.202966    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmq4\" (UniqueName: \"kubernetes.io/projected/166e94e3-43f7-42e2-9f02-0758789334ed-kube-api-access-fkmq4\") pod \"console-d855b574f-m2t5m\" (UID: \"166e94e3-43f7-42e2-9f02-0758789334ed\") " pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.222832 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.222801    6167 generic.go:356] "Generic (PLEG): container finished" podID="ab62d775-99b2-42a7-b44d-f24523284591" containerID="be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962" exitCode=0
Nov 03 17:46:21.222940 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.222845    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6797c4cb4d-zx29b" event={"ID":"ab62d775-99b2-42a7-b44d-f24523284591","Type":"ContainerDied","Data":"be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962"}
Nov 03 17:46:21.222940 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.222853    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6797c4cb4d-zx29b"
Nov 03 17:46:21.222940 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.222881    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6797c4cb4d-zx29b" event={"ID":"ab62d775-99b2-42a7-b44d-f24523284591","Type":"ContainerDied","Data":"626712c04812cf8011857d7352e6465d28b771533e91f88b1b30888f3dbb9669"}
Nov 03 17:46:21.222940 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.222901    6167 scope.go:122] "RemoveContainer" containerID="be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962"
Nov 03 17:46:21.230905 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.230881    6167 scope.go:122] "RemoveContainer" containerID="be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962"
Nov 03 17:46:21.231171 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:21.231154    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962\": container with ID starting with be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962 not found: ID does not exist" containerID="be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962"
Nov 03 17:46:21.231222 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.231180    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962"} err="failed to get container status \"be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962\": rpc error: code = NotFound desc = could not find container \"be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962\": container with ID starting with be62058e71feb874cb0f1fff64990bebbfe778c7bb5f33f19b39a76117a64962 not found: ID does not exist"
Nov 03 17:46:21.238390 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.238373    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:21.389020 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.388989    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6797c4cb4d-zx29b"]
Nov 03 17:46:21.393069 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.393045    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6797c4cb4d-zx29b"]
Nov 03 17:46:21.633738 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:46:21.633693    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166e94e3_43f7_42e2_9f02_0758789334ed.slice/crio-17cb4ddb5645474402e3bd8b9e99ab98b88dade17a236ea110a523ab44b99c48 WatchSource:0}: Error finding container 17cb4ddb5645474402e3bd8b9e99ab98b88dade17a236ea110a523ab44b99c48: Status 404 returned error can't find the container with id 17cb4ddb5645474402e3bd8b9e99ab98b88dade17a236ea110a523ab44b99c48
Nov 03 17:46:21.635013 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:21.634990    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d855b574f-m2t5m"]
Nov 03 17:46:22.231203 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:22.229922    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d855b574f-m2t5m" event={"ID":"166e94e3-43f7-42e2-9f02-0758789334ed","Type":"ContainerStarted","Data":"8116c6c88cf6947dd7d55de66cab84fa213ddb40a7c004ebeee4c5a65792c911"}
Nov 03 17:46:22.231203 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:22.230034    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d855b574f-m2t5m" event={"ID":"166e94e3-43f7-42e2-9f02-0758789334ed","Type":"ContainerStarted","Data":"17cb4ddb5645474402e3bd8b9e99ab98b88dade17a236ea110a523ab44b99c48"}
Nov 03 17:46:22.358562 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:22.358508    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d855b574f-m2t5m" podStartSLOduration=2.358494988 podStartE2EDuration="2.358494988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:22.358287239 +0000 UTC m=+37.588612311" watchObservedRunningTime="2026-11-03 17:46:22.358494988 +0000 UTC m=+37.588820060"
Nov 03 17:46:22.926228 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:22.926194    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab62d775-99b2-42a7-b44d-f24523284591" path="/var/lib/kubelet/pods/ab62d775-99b2-42a7-b44d-f24523284591/volumes"
Nov 03 17:46:24.519504 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:24.519468    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:46:24.560623 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.560581    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-574b5dd6dd-f4459" podUID="fc23c25a-4638-4c06-8804-3b1fe5135d56" containerName="console" containerID="cri-o://685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e" gracePeriod=15
Nov 03 17:46:24.869547 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.869519    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:46:24.974774 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.974742    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.974929 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.974806    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.974929 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.974849    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.975066 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.974947    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.975066 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.974979    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.975066 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.975014    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.975066 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.975060    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") pod \"fc23c25a-4638-4c06-8804-3b1fe5135d56\" (UID: \"fc23c25a-4638-4c06-8804-3b1fe5135d56\") "
Nov 03 17:46:24.975301 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.975269    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:24.975360 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.975323    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:24.975487 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.975462    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:24.975524 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.975473    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:24.977046 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.977024    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:24.977206 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.977186    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:24.977244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:24.977191    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7" pod "fc23c25a-4638-4c06-8804-3b1fe5135d56" (UID: "fc23c25a-4638-4c06-8804-3b1fe5135d56"). InnerVolumeSpecName "kube-api-access-px9k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076026    6167 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc23c25a-4638-4c06-8804-3b1fe5135d56-tmp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076049    6167 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-oauth-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076060    6167 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-oauth-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076069    6167 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076079    6167 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076088    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9k7\" (UniqueName: \"kubernetes.io/projected/fc23c25a-4638-4c06-8804-3b1fe5135d56-kube-api-access-px9k7\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.076093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.076097    6167 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc23c25a-4638-4c06-8804-3b1fe5135d56-console-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:25.245103 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.245060    6167 generic.go:356] "Generic (PLEG): container finished" podID="fc23c25a-4638-4c06-8804-3b1fe5135d56" containerID="685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e" exitCode=0
Nov 03 17:46:25.245252 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.245115    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574b5dd6dd-f4459" event={"ID":"fc23c25a-4638-4c06-8804-3b1fe5135d56","Type":"ContainerDied","Data":"685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e"}
Nov 03 17:46:25.245252 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.245127    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574b5dd6dd-f4459"
Nov 03 17:46:25.245252 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.245195    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574b5dd6dd-f4459" event={"ID":"fc23c25a-4638-4c06-8804-3b1fe5135d56","Type":"ContainerDied","Data":"3d0a9758e7075cbb07b8093af64c944bdfad4609e131466eb751899fd7ad8a95"}
Nov 03 17:46:25.245252 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.245215    6167 scope.go:122] "RemoveContainer" containerID="685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e"
Nov 03 17:46:25.254103 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.254067    6167 scope.go:122] "RemoveContainer" containerID="685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e"
Nov 03 17:46:25.254343 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:25.254317    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e\": container with ID starting with 685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e not found: ID does not exist" containerID="685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e"
Nov 03 17:46:25.254424 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.254353    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e"} err="failed to get container status \"685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e\": rpc error: code = NotFound desc = could not find container \"685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e\": container with ID starting with 685150349c44a648adb10b2cd653f0f42ef02367e406a3d7bde1114384b09b2e not found: ID does not exist"
Nov 03 17:46:25.410155 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.410076    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574b5dd6dd-f4459"]
Nov 03 17:46:25.414177 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:25.414156    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-574b5dd6dd-f4459"]
Nov 03 17:46:26.925963 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:26.925930    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc23c25a-4638-4c06-8804-3b1fe5135d56" path="/var/lib/kubelet/pods/fc23c25a-4638-4c06-8804-3b1fe5135d56/volumes"
Nov 03 17:46:31.239242 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:31.239212    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:31.239628 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:31.239264    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:31.243756 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:31.243732    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:31.274934 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:31.274914    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d855b574f-m2t5m"
Nov 03 17:46:31.513737 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:31.513631    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79b9d75875-9ztzl"]
Nov 03 17:46:35.340662 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.340601    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6db7d59f8-xrzpx" podUID="0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" containerName="console" containerID="cri-o://d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267" gracePeriod=15
Nov 03 17:46:35.597475 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.597422    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:46:35.666593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.666565    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.666750 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.666623    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.666750 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.666654    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2\" (UniqueName: \"kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.666750 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.666700    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.666939 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.666913    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:35.666996 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.666916    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.667055 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667009    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:35.667055 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667031    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.667178 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667077    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.667178 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667096    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle\") pod \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\" (UID: \"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98\") "
Nov 03 17:46:35.667374 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667324    6167 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.667374 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667347    6167 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-tmp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.667498 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667418    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:35.667498 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667425    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:35.667609 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.667495    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:35.668871 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.668849    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "kube-api-access-lcmv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:35.668871 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.668866    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:35.669013 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.668937    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config" pod "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" (UID: "0b62eb72-4187-4eb5-bb57-e48fdc9d5b98"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:35.768266 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.768234    6167 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-oauth-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.768266 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.768261    6167 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-oauth-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.768266 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.768272    6167 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.768485 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.768282    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-trusted-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.768485 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.768292    6167 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-console-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:35.768485 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:35.768300    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcmv2\" (UniqueName: \"kubernetes.io/projected/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98-kube-api-access-lcmv2\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:36.293370 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.293331    6167 generic.go:356] "Generic (PLEG): container finished" podID="0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" containerID="d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267" exitCode=0
Nov 03 17:46:36.293554 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.293385    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db7d59f8-xrzpx"
Nov 03 17:46:36.293554 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.293419    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db7d59f8-xrzpx" event={"ID":"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98","Type":"ContainerDied","Data":"d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267"}
Nov 03 17:46:36.293554 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.293465    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db7d59f8-xrzpx" event={"ID":"0b62eb72-4187-4eb5-bb57-e48fdc9d5b98","Type":"ContainerDied","Data":"029ec813e2355e0c58d6b795f1cef5380a6bdbf2e26ab326f9649d7d48cffd0c"}
Nov 03 17:46:36.293554 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.293484    6167 scope.go:122] "RemoveContainer" containerID="d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267"
Nov 03 17:46:36.302588 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.302554    6167 scope.go:122] "RemoveContainer" containerID="d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267"
Nov 03 17:46:36.302877 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:36.302857    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267\": container with ID starting with d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267 not found: ID does not exist" containerID="d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267"
Nov 03 17:46:36.302932 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.302889    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267"} err="failed to get container status \"d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267\": rpc error: code = NotFound desc = could not find container \"d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267\": container with ID starting with d7bef20b8f1ee50f5f3746e4f1f2aa649bcc62de287a2fda2bae27df28280267 not found: ID does not exist"
Nov 03 17:46:36.458922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.458886    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db7d59f8-xrzpx"]
Nov 03 17:46:36.509257 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.509225    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6db7d59f8-xrzpx"]
Nov 03 17:46:36.925882 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:36.925850    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" path="/var/lib/kubelet/pods/0b62eb72-4187-4eb5-bb57-e48fdc9d5b98/volumes"
Nov 03 17:46:54.519339 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:54.519303    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:46:56.533161 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.533121    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79b9d75875-9ztzl" podUID="dad54663-9592-4ef3-865e-d92b39fc946b" containerName="console" containerID="cri-o://ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564" gracePeriod=15
Nov 03 17:46:56.814404 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.814345    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:56.953260 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953229    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953447 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953280    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953447 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953395    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953447 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953442    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953476    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg\" (UniqueName: \"kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953505    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953567    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953618 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953598    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca\") pod \"dad54663-9592-4ef3-865e-d92b39fc946b\" (UID: \"dad54663-9592-4ef3-865e-d92b39fc946b\") "
Nov 03 17:46:56.953804 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953668    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:46:56.953804 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953694    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:56.953988 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953968    6167 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-oauth-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:56.954082 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.953984    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:56.954082 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.954001    6167 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad54663-9592-4ef3-865e-d92b39fc946b-tmp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:56.954082 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.954037    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:56.954267 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.954143    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:46:56.955516 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.955490    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:56.955984 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.955950    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "kube-api-access-w76pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:46:56.955984 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:56.955967    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config" pod "dad54663-9592-4ef3-865e-d92b39fc946b" (UID: "dad54663-9592-4ef3-865e-d92b39fc946b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:46:57.055444 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.055413    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w76pg\" (UniqueName: \"kubernetes.io/projected/dad54663-9592-4ef3-865e-d92b39fc946b-kube-api-access-w76pg\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.055444 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.055438    6167 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-oauth-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.055444 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.055448    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-trusted-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.055678 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.055457    6167 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.055678 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.055466    6167 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dad54663-9592-4ef3-865e-d92b39fc946b-console-serving-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.055678 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.055474    6167 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dad54663-9592-4ef3-865e-d92b39fc946b-console-config\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:46:57.379622 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.379579    6167 generic.go:356] "Generic (PLEG): container finished" podID="dad54663-9592-4ef3-865e-d92b39fc946b" containerID="ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564" exitCode=0
Nov 03 17:46:57.379622 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.379623    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9d75875-9ztzl"
Nov 03 17:46:57.379844 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.379659    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9d75875-9ztzl" event={"ID":"dad54663-9592-4ef3-865e-d92b39fc946b","Type":"ContainerDied","Data":"ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564"}
Nov 03 17:46:57.379844 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.379707    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9d75875-9ztzl" event={"ID":"dad54663-9592-4ef3-865e-d92b39fc946b","Type":"ContainerDied","Data":"72df771456ec7f497ae82f418ec0c6f96912f11aff8d8cda491cd0c3aefec3b6"}
Nov 03 17:46:57.379844 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.379742    6167 scope.go:122] "RemoveContainer" containerID="ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564"
Nov 03 17:46:57.388557 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.388541    6167 scope.go:122] "RemoveContainer" containerID="ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564"
Nov 03 17:46:57.388837 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:46:57.388819    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564\": container with ID starting with ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564 not found: ID does not exist" containerID="ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564"
Nov 03 17:46:57.388899 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.388862    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564"} err="failed to get container status \"ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564\": rpc error: code = NotFound desc = could not find container \"ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564\": container with ID starting with ac444ec57f5734c7ae4886a52f04441f35b198ae01d909367c81ce5e9777a564 not found: ID does not exist"
Nov 03 17:46:57.544553 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.544521    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79b9d75875-9ztzl"]
Nov 03 17:46:57.595171 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:57.595139    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79b9d75875-9ztzl"]
Nov 03 17:46:58.925107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:46:58.925074    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad54663-9592-4ef3-865e-d92b39fc946b" path="/var/lib/kubelet/pods/dad54663-9592-4ef3-865e-d92b39fc946b/volumes"
Nov 03 17:47:24.518708 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:47:24.518629    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:47:54.519006 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:47:54.518967    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:48:24.519615 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:48:24.519568    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:48:54.520846 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:48:54.520757    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:48:57.825252 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825218    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"]
Nov 03 17:48:57.825709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825592    6167 cpu_manager.go:414] "containerMap: removing container" podUID="0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" containerName="console"
Nov 03 17:48:57.825709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825605    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" containerName="console"
Nov 03 17:48:57.825709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825633    6167 cpu_manager.go:414] "containerMap: removing container" podUID="fc23c25a-4638-4c06-8804-3b1fe5135d56" containerName="console"
Nov 03 17:48:57.825709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825638    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc23c25a-4638-4c06-8804-3b1fe5135d56" containerName="console"
Nov 03 17:48:57.825709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825657    6167 cpu_manager.go:414] "containerMap: removing container" podUID="dad54663-9592-4ef3-865e-d92b39fc946b" containerName="console"
Nov 03 17:48:57.825709 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825662    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad54663-9592-4ef3-865e-d92b39fc946b" containerName="console"
Nov 03 17:48:57.825927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825746    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad54663-9592-4ef3-865e-d92b39fc946b" containerName="console"
Nov 03 17:48:57.825927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825756    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b62eb72-4187-4eb5-bb57-e48fdc9d5b98" containerName="console"
Nov 03 17:48:57.825927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.825764    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc23c25a-4638-4c06-8804-3b1fe5135d56" containerName="console"
Nov 03 17:48:57.827694 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.827679    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:57.929066 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.929032    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:57.929238 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.929083    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdqf\" (UniqueName: \"kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:57.929238 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.929167    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:57.931240 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.931223    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 17:48:57.931292 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.931226    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kdsk5\""
Nov 03 17:48:57.931355 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.931339    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 17:48:57.932148 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:57.932132    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"]
Nov 03 17:48:58.030123 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.030095    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.030284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.030161    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.030284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.030197    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdqf\" (UniqueName: \"kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.030457 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.030439    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.030500 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.030472    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.052492 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.052466    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"]
Nov 03 17:48:58.054650 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.054635    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.092456 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.092391    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdqf\" (UniqueName: \"kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.107907 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.107885    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z9hfg\""
Nov 03 17:48:58.112550 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.112527    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"]
Nov 03 17:48:58.131133 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.131112    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.131239 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.131169    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.131279 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.131243    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpf7\" (UniqueName: \"kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.140006 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.139987    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:48:58.232326 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.232296    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.232505 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.232351    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.232505 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.232370    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpf7\" (UniqueName: \"kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.232689 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.232669    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.232772 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.232742    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.303491 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.303464    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpf7\" (UniqueName: \"kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.366756 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.366655    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:48:58.505578 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:48:58.505548    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fb05d1_03e1_4278_9dfc_a214d63d4152.slice/crio-fe09e748219ac558e86be506b976533f5ad034973239a0b41d3b98734cce39de WatchSource:0}: Error finding container fe09e748219ac558e86be506b976533f5ad034973239a0b41d3b98734cce39de: Status 404 returned error can't find the container with id fe09e748219ac558e86be506b976533f5ad034973239a0b41d3b98734cce39de
Nov 03 17:48:58.507418 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.507390    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"]
Nov 03 17:48:58.733421 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:48:58.733390    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1d35d2_ae95_4d32_b3d6_9e7dd5ee172a.slice/crio-90ce6b39a5afabc6a1016d1de0f6450fbd878c6b5658bd5b1a6eca1294ef921c WatchSource:0}: Error finding container 90ce6b39a5afabc6a1016d1de0f6450fbd878c6b5658bd5b1a6eca1294ef921c: Status 404 returned error can't find the container with id 90ce6b39a5afabc6a1016d1de0f6450fbd878c6b5658bd5b1a6eca1294ef921c
Nov 03 17:48:58.734400 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.734376    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"]
Nov 03 17:48:58.850226 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.850186    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq" event={"ID":"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a","Type":"ContainerStarted","Data":"90ce6b39a5afabc6a1016d1de0f6450fbd878c6b5658bd5b1a6eca1294ef921c"}
Nov 03 17:48:58.851404 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:58.851382    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp" event={"ID":"c3fb05d1-03e1-4278-9dfc-a214d63d4152","Type":"ContainerStarted","Data":"fe09e748219ac558e86be506b976533f5ad034973239a0b41d3b98734cce39de"}
Nov 03 17:48:59.375768 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.375735    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"]
Nov 03 17:48:59.378864 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.378845    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.435202 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.435170    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-s42jz\""
Nov 03 17:48:59.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.442500    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.442679 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.442576    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.442679 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.442647    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2d7t\" (UniqueName: \"kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.486593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.486560    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"]
Nov 03 17:48:59.544006 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.543969    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.544190 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.544033    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.544190 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.544067    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d7t\" (UniqueName: \"kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.544482 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.544451    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.544594 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.544513    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.606634 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.606604    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d7t\" (UniqueName: \"kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:48:59.691911 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:48:59.691837    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:49:00.080441 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:00.080402    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"]
Nov 03 17:49:00.083865 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:00.083832    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a02b55_5c0b_49ef_800f_83d92313083a.slice/crio-26d9e44566d41a6d5687888fd9075c110dc2e8bd746f3cb68fc3bf0baab6dcf3 WatchSource:0}: Error finding container 26d9e44566d41a6d5687888fd9075c110dc2e8bd746f3cb68fc3bf0baab6dcf3: Status 404 returned error can't find the container with id 26d9e44566d41a6d5687888fd9075c110dc2e8bd746f3cb68fc3bf0baab6dcf3
Nov 03 17:49:00.865024 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:00.864986    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2" event={"ID":"96a02b55-5c0b-49ef-800f-83d92313083a","Type":"ContainerStarted","Data":"26d9e44566d41a6d5687888fd9075c110dc2e8bd746f3cb68fc3bf0baab6dcf3"}
Nov 03 17:49:01.174824 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.174739    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"]
Nov 03 17:49:01.179110 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.178556    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.234155 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.234104    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 17:49:01.258853 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.258822    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h54vm\""
Nov 03 17:49:01.261626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261550    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.261626 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261601    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.261855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261739    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.261855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261787    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.261855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261833    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.262055 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261870    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.262055 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261907    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.262055 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261942    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.262055 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.261977    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplxt\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.285787 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.285697    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"]
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.362810    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.362859    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.362906    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.362934    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.362968    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363003    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363032    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363064    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363100    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplxt\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363223    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.363695 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363653    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.364400 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.363967    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.364400 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.364148    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.364510 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.364400    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.367639 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.367563    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.368932 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.368907    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.425564 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.425487    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.426643 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.426616    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplxt\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt\") pod \"maas-default-gateway-openshift-default-6dcf757899-r597d\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.494521 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.494482    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:01.956877 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:01.956840    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"]
Nov 03 17:49:01.983599 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:01.983565    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b57a48_4012_4b73_9a5c_e69fde62eaf7.slice/crio-4333a7e167ef0d16cb7c41b67acc249fce6c882b9cd52312241c58a20c660b47 WatchSource:0}: Error finding container 4333a7e167ef0d16cb7c41b67acc249fce6c882b9cd52312241c58a20c660b47: Status 404 returned error can't find the container with id 4333a7e167ef0d16cb7c41b67acc249fce6c882b9cd52312241c58a20c660b47
Nov 03 17:49:02.879361 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:02.879320    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" event={"ID":"87b57a48-4012-4b73-9a5c-e69fde62eaf7","Type":"ContainerStarted","Data":"4333a7e167ef0d16cb7c41b67acc249fce6c882b9cd52312241c58a20c660b47"}
Nov 03 17:49:06.903890 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:06.903860    6167 generic.go:356] "Generic (PLEG): container finished" podID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerID="ae6c672f8b70bbed10cdb598f8c34edb13e605941479db0b438a61ada49d2537" exitCode=0
Nov 03 17:49:06.904380 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:06.903949    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp" event={"ID":"c3fb05d1-03e1-4278-9dfc-a214d63d4152","Type":"ContainerDied","Data":"ae6c672f8b70bbed10cdb598f8c34edb13e605941479db0b438a61ada49d2537"}
Nov 03 17:49:07.912826 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:07.912793    6167 generic.go:356] "Generic (PLEG): container finished" podID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerID="a5e7bd6c472ba23ba220cd68786846cac13c14f4700f969ac19d213e96b505ff" exitCode=0
Nov 03 17:49:07.913225 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:07.912884    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq" event={"ID":"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a","Type":"ContainerDied","Data":"a5e7bd6c472ba23ba220cd68786846cac13c14f4700f969ac19d213e96b505ff"}
Nov 03 17:49:08.606347 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:08.606310    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:49:08.606489 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:08.606381    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:49:08.606489 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:08.606415    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:49:08.918641 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:08.918557    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" event={"ID":"87b57a48-4012-4b73-9a5c-e69fde62eaf7","Type":"ContainerStarted","Data":"037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"}
Nov 03 17:49:09.140616 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.140565    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podStartSLOduration=1.521730712 podStartE2EDuration="8.140549286s" totalImagesPullingTime="6.618818574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:09.090077362 +0000 UTC m=+204.320402433" watchObservedRunningTime="2026-11-03 17:49:09.140549286 +0000 UTC m=+204.370874357"
Nov 03 17:49:09.495301 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.495223    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:09.500682 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.500656    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:09.925500 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.925460    6167 generic.go:356] "Generic (PLEG): container finished" podID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerID="a505905abaa9a27a38c5d521c892daf2104e35c09cd9d1737aa703145c6c1d03" exitCode=0
Nov 03 17:49:09.926042 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.925802    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp" event={"ID":"c3fb05d1-03e1-4278-9dfc-a214d63d4152","Type":"ContainerDied","Data":"a505905abaa9a27a38c5d521c892daf2104e35c09cd9d1737aa703145c6c1d03"}
Nov 03 17:49:09.927804 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.927755    6167 generic.go:356] "Generic (PLEG): container finished" podID="96a02b55-5c0b-49ef-800f-83d92313083a" containerID="3b5e0decb6b943fa8725ad3c1f5dbe6853b24ffaa488c13f2e2bf16756b351f9" exitCode=0
Nov 03 17:49:09.927870 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.927837    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2" event={"ID":"96a02b55-5c0b-49ef-800f-83d92313083a","Type":"ContainerDied","Data":"3b5e0decb6b943fa8725ad3c1f5dbe6853b24ffaa488c13f2e2bf16756b351f9"}
Nov 03 17:49:09.928145 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.928115    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:09.929312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:09.929295    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:49:10.934507 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:10.934470    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq" event={"ID":"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a","Type":"ContainerStarted","Data":"af9732b21f251be8e763255e1ebdfa2cae2e42317927547acf777e63b0477ee1"}
Nov 03 17:49:11.941636 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:11.941601    6167 generic.go:356] "Generic (PLEG): container finished" podID="96a02b55-5c0b-49ef-800f-83d92313083a" containerID="45e3e5d44e807ba0a9210803f21c209553bc376639601f652dd5c44e4f58f9ae" exitCode=0
Nov 03 17:49:11.942122 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:11.941733    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2" event={"ID":"96a02b55-5c0b-49ef-800f-83d92313083a","Type":"ContainerDied","Data":"45e3e5d44e807ba0a9210803f21c209553bc376639601f652dd5c44e4f58f9ae"}
Nov 03 17:49:11.944130 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:11.944101    6167 generic.go:356] "Generic (PLEG): container finished" podID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerID="af9732b21f251be8e763255e1ebdfa2cae2e42317927547acf777e63b0477ee1" exitCode=0
Nov 03 17:49:11.944396 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:11.944235    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq" event={"ID":"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a","Type":"ContainerDied","Data":"af9732b21f251be8e763255e1ebdfa2cae2e42317927547acf777e63b0477ee1"}
Nov 03 17:49:18.999398 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:18.999308    6167 generic.go:356] "Generic (PLEG): container finished" podID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerID="1f712def40e5ff11498da06c631610eb0acf4ba27fd9825b59e36f4507cd8fd1" exitCode=0
Nov 03 17:49:18.999398 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:18.999378    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp" event={"ID":"c3fb05d1-03e1-4278-9dfc-a214d63d4152","Type":"ContainerDied","Data":"1f712def40e5ff11498da06c631610eb0acf4ba27fd9825b59e36f4507cd8fd1"}
Nov 03 17:49:20.136167 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.136140    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:49:20.153585 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.153560    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf\" (UniqueName: \"kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf\") pod \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") "
Nov 03 17:49:20.153698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.153658    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util\") pod \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") "
Nov 03 17:49:20.153698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.153682    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle\") pod \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\" (UID: \"c3fb05d1-03e1-4278-9dfc-a214d63d4152\") "
Nov 03 17:49:20.154345 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.154318    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle" pod "c3fb05d1-03e1-4278-9dfc-a214d63d4152" (UID: "c3fb05d1-03e1-4278-9dfc-a214d63d4152"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:20.155728 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.155699    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf" pod "c3fb05d1-03e1-4278-9dfc-a214d63d4152" (UID: "c3fb05d1-03e1-4278-9dfc-a214d63d4152"). InnerVolumeSpecName "kube-api-access-skdqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:20.157472 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.157447    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util" pod "c3fb05d1-03e1-4278-9dfc-a214d63d4152" (UID: "c3fb05d1-03e1-4278-9dfc-a214d63d4152"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:20.254601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.254538    6167 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-util\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:20.254601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.254563    6167 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fb05d1-03e1-4278-9dfc-a214d63d4152-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:20.254601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:20.254578    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdqf\" (UniqueName: \"kubernetes.io/projected/c3fb05d1-03e1-4278-9dfc-a214d63d4152-kube-api-access-skdqf\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:21.011356 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.011319    6167 generic.go:356] "Generic (PLEG): container finished" podID="96a02b55-5c0b-49ef-800f-83d92313083a" containerID="028bc342b70d8b10d56a986ef54e753363cfb495fc65aacef8927855060e6b6c" exitCode=0
Nov 03 17:49:21.011502 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.011394    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2" event={"ID":"96a02b55-5c0b-49ef-800f-83d92313083a","Type":"ContainerDied","Data":"028bc342b70d8b10d56a986ef54e753363cfb495fc65aacef8927855060e6b6c"}
Nov 03 17:49:21.013407 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.013385    6167 generic.go:356] "Generic (PLEG): container finished" podID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerID="d00a6e2fc6bc17cbc6f12a2f2bd6df24fd81d586dc44d8c25fa24a81d2f97035" exitCode=0
Nov 03 17:49:21.013491 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.013456    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq" event={"ID":"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a","Type":"ContainerDied","Data":"d00a6e2fc6bc17cbc6f12a2f2bd6df24fd81d586dc44d8c25fa24a81d2f97035"}
Nov 03 17:49:21.015214 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.015192    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp" event={"ID":"c3fb05d1-03e1-4278-9dfc-a214d63d4152","Type":"ContainerDied","Data":"fe09e748219ac558e86be506b976533f5ad034973239a0b41d3b98734cce39de"}
Nov 03 17:49:21.015296 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.015222    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6dfgp"
Nov 03 17:49:21.015296 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:21.015231    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe09e748219ac558e86be506b976533f5ad034973239a0b41d3b98734cce39de"
Nov 03 17:49:22.143603 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.143578    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:49:22.170615 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.170591    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util\") pod \"96a02b55-5c0b-49ef-800f-83d92313083a\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") "
Nov 03 17:49:22.170734 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.170666    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t\" (UniqueName: \"kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t\") pod \"96a02b55-5c0b-49ef-800f-83d92313083a\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") "
Nov 03 17:49:22.170734 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.170693    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle\") pod \"96a02b55-5c0b-49ef-800f-83d92313083a\" (UID: \"96a02b55-5c0b-49ef-800f-83d92313083a\") "
Nov 03 17:49:22.171377 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.171345    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle" pod "96a02b55-5c0b-49ef-800f-83d92313083a" (UID: "96a02b55-5c0b-49ef-800f-83d92313083a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:22.172909 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.172879    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t" pod "96a02b55-5c0b-49ef-800f-83d92313083a" (UID: "96a02b55-5c0b-49ef-800f-83d92313083a"). InnerVolumeSpecName "kube-api-access-x2d7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:22.175833 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.175811    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util" pod "96a02b55-5c0b-49ef-800f-83d92313083a" (UID: "96a02b55-5c0b-49ef-800f-83d92313083a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:22.177589 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.177572    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:49:22.272158 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272085    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util\") pod \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") "
Nov 03 17:49:22.272158 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272135    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle\") pod \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") "
Nov 03 17:49:22.272376 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272197    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7\" (UniqueName: \"kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7\") pod \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\" (UID: \"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a\") "
Nov 03 17:49:22.272376 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272358    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2d7t\" (UniqueName: \"kubernetes.io/projected/96a02b55-5c0b-49ef-800f-83d92313083a-kube-api-access-x2d7t\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:22.272376 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272371    6167 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:22.272548 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272381    6167 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a02b55-5c0b-49ef-800f-83d92313083a-util\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:22.272870 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.272844    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle" pod "ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" (UID: "ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:22.274327 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.274299    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7" pod "ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" (UID: "ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a"). InnerVolumeSpecName "kube-api-access-4xpf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:49:22.277282 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.277252    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util" pod "ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" (UID: "ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:49:22.373792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.373757    6167 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:22.373792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.373785    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpf7\" (UniqueName: \"kubernetes.io/projected/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-kube-api-access-4xpf7\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:22.373792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:22.373794    6167 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a-util\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:49:23.025446 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:23.025415    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2"
Nov 03 17:49:23.025596 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:23.025414    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712f5t2" event={"ID":"96a02b55-5c0b-49ef-800f-83d92313083a","Type":"ContainerDied","Data":"26d9e44566d41a6d5687888fd9075c110dc2e8bd746f3cb68fc3bf0baab6dcf3"}
Nov 03 17:49:23.025596 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:23.025552    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d9e44566d41a6d5687888fd9075c110dc2e8bd746f3cb68fc3bf0baab6dcf3"
Nov 03 17:49:23.027627 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:23.027597    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq" event={"ID":"ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a","Type":"ContainerDied","Data":"90ce6b39a5afabc6a1016d1de0f6450fbd878c6b5658bd5b1a6eca1294ef921c"}
Nov 03 17:49:23.027778 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:23.027641    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ce6b39a5afabc6a1016d1de0f6450fbd878c6b5658bd5b1a6eca1294ef921c"
Nov 03 17:49:23.027778 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:23.027651    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7l7wq"
Nov 03 17:49:24.519960 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:49:24.519914    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:49:26.910137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910097    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"]
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910564    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="pull"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910582    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="pull"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910604    6167 cpu_manager.go:414] "containerMap: removing container" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="util"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910613    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="util"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910628    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="util"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910635    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="util"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910656    6167 cpu_manager.go:414] "containerMap: removing container" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="extract"
Nov 03 17:49:26.910667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910663    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910676    6167 cpu_manager.go:414] "containerMap: removing container" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910684    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910697    6167 cpu_manager.go:414] "containerMap: removing container" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="pull"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910704    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="pull"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910739    6167 cpu_manager.go:414] "containerMap: removing container" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="pull"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910746    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="pull"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910757    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910764    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910779    6167 cpu_manager.go:414] "containerMap: removing container" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="util"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910786    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="util"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910883    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fb05d1-03e1-4278-9dfc-a214d63d4152" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910896    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a02b55-5c0b-49ef-800f-83d92313083a" containerName="extract"
Nov 03 17:49:26.911087 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.910906    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1d35d2-ae95-4d32-b3d6-9e7dd5ee172a" containerName="extract"
Nov 03 17:49:26.990142 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.990115    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:26.997165 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:26.997139    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"]
Nov 03 17:49:27.009965 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.009943    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/789f1435-8a1e-42e1-8ffc-5bf0a9514b0e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk\" (UID: \"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.010080 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.010054    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtvpj\" (UniqueName: \"kubernetes.io/projected/789f1435-8a1e-42e1-8ffc-5bf0a9514b0e-kube-api-access-dtvpj\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk\" (UID: \"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.045975 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.045949    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 17:49:27.045975 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.045949    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:49:27.046144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.045981    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fphxk\""
Nov 03 17:49:27.110688 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.110663    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtvpj\" (UniqueName: \"kubernetes.io/projected/789f1435-8a1e-42e1-8ffc-5bf0a9514b0e-kube-api-access-dtvpj\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk\" (UID: \"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.110827 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.110703    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/789f1435-8a1e-42e1-8ffc-5bf0a9514b0e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk\" (UID: \"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.111024 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.111008    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/789f1435-8a1e-42e1-8ffc-5bf0a9514b0e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk\" (UID: \"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.180955 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.180904    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtvpj\" (UniqueName: \"kubernetes.io/projected/789f1435-8a1e-42e1-8ffc-5bf0a9514b0e-kube-api-access-dtvpj\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk\" (UID: \"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.323485 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.323441    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"
Nov 03 17:49:27.807276 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:27.807243    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789f1435_8a1e_42e1_8ffc_5bf0a9514b0e.slice/crio-21c467d8a37c5c2b52d500ffa369b5567ff3f0560e4df9fcc632e534de5909b6 WatchSource:0}: Error finding container 21c467d8a37c5c2b52d500ffa369b5567ff3f0560e4df9fcc632e534de5909b6: Status 404 returned error can't find the container with id 21c467d8a37c5c2b52d500ffa369b5567ff3f0560e4df9fcc632e534de5909b6
Nov 03 17:49:27.808305 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:27.808280    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk"]
Nov 03 17:49:28.050234 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:28.050201    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk" event={"ID":"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e","Type":"ContainerStarted","Data":"21c467d8a37c5c2b52d500ffa369b5567ff3f0560e4df9fcc632e534de5909b6"}
Nov 03 17:49:31.065643 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:31.065605    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk" event={"ID":"789f1435-8a1e-42e1-8ffc-5bf0a9514b0e","Type":"ContainerStarted","Data":"78af91ba5988b4119daef4aa259194a61b140c7ba1fd1aecef03b34f7e82d148"}
Nov 03 17:49:31.191574 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:31.191481    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bq7kk" podStartSLOduration=2.063697388 podStartE2EDuration="5.191467361s" totalImagesPullingTime="3.127769973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:31.190935224 +0000 UTC m=+226.421260295" watchObservedRunningTime="2026-11-03 17:49:31.191467361 +0000 UTC m=+226.421792426"
Nov 03 17:49:34.557863 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.557832    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"]
Nov 03 17:49:34.560103 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.560088    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:34.625200 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.625173    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 17:49:34.671683 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.671654    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 17:49:34.671857 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.671793    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-snvg6\""
Nov 03 17:49:34.672889 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.672871    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"]
Nov 03 17:49:34.673961 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.673942    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g42\" (UniqueName: \"kubernetes.io/projected/f91e3668-e29d-438e-964a-1fe7eba2fe02-kube-api-access-r7g42\") pod \"limitador-operator-controller-manager-5778bb4449-fss7p\" (UID: \"f91e3668-e29d-438e-964a-1fe7eba2fe02\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:34.775240 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.775202    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g42\" (UniqueName: \"kubernetes.io/projected/f91e3668-e29d-438e-964a-1fe7eba2fe02-kube-api-access-r7g42\") pod \"limitador-operator-controller-manager-5778bb4449-fss7p\" (UID: \"f91e3668-e29d-438e-964a-1fe7eba2fe02\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:34.862199 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.862173    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g42\" (UniqueName: \"kubernetes.io/projected/f91e3668-e29d-438e-964a-1fe7eba2fe02-kube-api-access-r7g42\") pod \"limitador-operator-controller-manager-5778bb4449-fss7p\" (UID: \"f91e3668-e29d-438e-964a-1fe7eba2fe02\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:34.871453 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:34.871430    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:35.247453 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:35.247421    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91e3668_e29d_438e_964a_1fe7eba2fe02.slice/crio-4aa1ae137b941bffe2da17398f521a35f586a47cbcb65d00efe2c8263a4743c8 WatchSource:0}: Error finding container 4aa1ae137b941bffe2da17398f521a35f586a47cbcb65d00efe2c8263a4743c8: Status 404 returned error can't find the container with id 4aa1ae137b941bffe2da17398f521a35f586a47cbcb65d00efe2c8263a4743c8
Nov 03 17:49:35.248088 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:35.248064    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"]
Nov 03 17:49:36.087938 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:36.087903    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p" event={"ID":"f91e3668-e29d-438e-964a-1fe7eba2fe02","Type":"ContainerStarted","Data":"4aa1ae137b941bffe2da17398f521a35f586a47cbcb65d00efe2c8263a4743c8"}
Nov 03 17:49:38.803012 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:38.802977    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"]
Nov 03 17:49:38.805071 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:38.805055    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:38.859849 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:38.859780    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 17:49:38.859983 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:38.859922    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f4rj9\""
Nov 03 17:49:38.909504 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:38.909478    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9lm\" (UniqueName: \"kubernetes.io/projected/1d5a2954-b796-4c83-9725-8fd5085c87b1-kube-api-access-6x9lm\") pod \"dns-operator-controller-manager-8557cbdb84-5xc7p\" (UID: \"1d5a2954-b796-4c83-9725-8fd5085c87b1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:38.912745 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:38.912705    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"]
Nov 03 17:49:39.010502 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.010473    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9lm\" (UniqueName: \"kubernetes.io/projected/1d5a2954-b796-4c83-9725-8fd5085c87b1-kube-api-access-6x9lm\") pod \"dns-operator-controller-manager-8557cbdb84-5xc7p\" (UID: \"1d5a2954-b796-4c83-9725-8fd5085c87b1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:39.072466 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.072427    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9lm\" (UniqueName: \"kubernetes.io/projected/1d5a2954-b796-4c83-9725-8fd5085c87b1-kube-api-access-6x9lm\") pod \"dns-operator-controller-manager-8557cbdb84-5xc7p\" (UID: \"1d5a2954-b796-4c83-9725-8fd5085c87b1\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:39.103681 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.103655    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p" event={"ID":"f91e3668-e29d-438e-964a-1fe7eba2fe02","Type":"ContainerStarted","Data":"8f14cd7487dacab6bd9551f2e498a15d3e6c64e6dced3ee1572d580045e6103e"}
Nov 03 17:49:39.103841 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.103791    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:39.116215 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.116172    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:39.245312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.245268    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p" podStartSLOduration=1.916894887 podStartE2EDuration="5.245254032s" totalImagesPullingTime="3.328359145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:39.244753887 +0000 UTC m=+234.475078956" watchObservedRunningTime="2026-11-03 17:49:39.245254032 +0000 UTC m=+234.475579104"
Nov 03 17:49:39.496276 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:39.496234    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5a2954_b796_4c83_9725_8fd5085c87b1.slice/crio-a56e57ba40d30382e1cd0ffa2eb41f366a78cfcfdb5e4f6402a33355b1fdb51e WatchSource:0}: Error finding container a56e57ba40d30382e1cd0ffa2eb41f366a78cfcfdb5e4f6402a33355b1fdb51e: Status 404 returned error can't find the container with id a56e57ba40d30382e1cd0ffa2eb41f366a78cfcfdb5e4f6402a33355b1fdb51e
Nov 03 17:49:39.496757 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:39.496712    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"]
Nov 03 17:49:40.115230 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.115193    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p" event={"ID":"1d5a2954-b796-4c83-9725-8fd5085c87b1","Type":"ContainerStarted","Data":"a56e57ba40d30382e1cd0ffa2eb41f366a78cfcfdb5e4f6402a33355b1fdb51e"}
Nov 03 17:49:40.249332 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.249301    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"]
Nov 03 17:49:40.252481 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.252462    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.322551 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.322521    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.322551 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.322554    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqv7\" (UniqueName: \"kubernetes.io/projected/b9731ed5-3173-443b-8992-51c6cfc38e6f-kube-api-access-5bqv7\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.322760 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.322584    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.322760 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.322682    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9731ed5-3173-443b-8992-51c6cfc38e6f-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.322760 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.322745    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9731ed5-3173-443b-8992-51c6cfc38e6f-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.322867 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.322798    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.362471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.362432    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 17:49:40.362471 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.362431    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 17:49:40.362710 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.362537    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 17:49:40.362710 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.362628    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 17:49:40.362710 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.362668    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 17:49:40.362710 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.362680    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jdbrd\""
Nov 03 17:49:40.363927 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.363901    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"]
Nov 03 17:49:40.423539 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.423447    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.423539 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.423487    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqv7\" (UniqueName: \"kubernetes.io/projected/b9731ed5-3173-443b-8992-51c6cfc38e6f-kube-api-access-5bqv7\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.423539 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.423534    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.423859 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.423571    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9731ed5-3173-443b-8992-51c6cfc38e6f-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.423859 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.423593    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9731ed5-3173-443b-8992-51c6cfc38e6f-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.423859 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.423633    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.424096 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.424065    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9731ed5-3173-443b-8992-51c6cfc38e6f-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.425894 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.425866    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9731ed5-3173-443b-8992-51c6cfc38e6f-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.426231 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.426205    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.426375 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.426357    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.426445 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.426404    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9731ed5-3173-443b-8992-51c6cfc38e6f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.522331 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.522295    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqv7\" (UniqueName: \"kubernetes.io/projected/b9731ed5-3173-443b-8992-51c6cfc38e6f-kube-api-access-5bqv7\") pod \"opendatahub-operator-controller-manager-6dd86677fd-kk49v\" (UID: \"b9731ed5-3173-443b-8992-51c6cfc38e6f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:40.565917 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:40.565881    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:41.090589 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:41.090553    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9731ed5_3173_443b_8992_51c6cfc38e6f.slice/crio-af177c7f52ef4b27a6e35ac88919aad96dabe67e54b57f0aadd8b7979f1f9fd6 WatchSource:0}: Error finding container af177c7f52ef4b27a6e35ac88919aad96dabe67e54b57f0aadd8b7979f1f9fd6: Status 404 returned error can't find the container with id af177c7f52ef4b27a6e35ac88919aad96dabe67e54b57f0aadd8b7979f1f9fd6
Nov 03 17:49:41.096752 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:41.096712    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"]
Nov 03 17:49:41.121613 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:41.121585    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v" event={"ID":"b9731ed5-3173-443b-8992-51c6cfc38e6f","Type":"ContainerStarted","Data":"af177c7f52ef4b27a6e35ac88919aad96dabe67e54b57f0aadd8b7979f1f9fd6"}
Nov 03 17:49:42.127734 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:42.127693    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p" event={"ID":"1d5a2954-b796-4c83-9725-8fd5085c87b1","Type":"ContainerStarted","Data":"dce7769047a445139c43ada624550513f1680207d38e5fbafd55a65c5c2405fb"}
Nov 03 17:49:42.128148 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:42.127796    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:42.260220 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:42.260123    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p" podStartSLOduration=1.823060777 podStartE2EDuration="4.26010749s" totalImagesPullingTime="2.437046713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:42.25933529 +0000 UTC m=+237.489660362" watchObservedRunningTime="2026-11-03 17:49:42.26010749 +0000 UTC m=+237.490432615"
Nov 03 17:49:44.486105 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.484577    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"]
Nov 03 17:49:44.491265 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.491243    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:49:44.557278 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.557205    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"]
Nov 03 17:49:44.564308 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.564280    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gdx\" (UniqueName: \"kubernetes.io/projected/00003c75-2730-4420-86db-53ebe728b1e5-kube-api-access-85gdx\") pod \"authorino-operator-7b7bb4bc77-kqnnj\" (UID: \"00003c75-2730-4420-86db-53ebe728b1e5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:49:44.606099 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.606060    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mjfc4\""
Nov 03 17:49:44.665975 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.665939    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gdx\" (UniqueName: \"kubernetes.io/projected/00003c75-2730-4420-86db-53ebe728b1e5-kube-api-access-85gdx\") pod \"authorino-operator-7b7bb4bc77-kqnnj\" (UID: \"00003c75-2730-4420-86db-53ebe728b1e5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:49:44.740267 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.740192    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gdx\" (UniqueName: \"kubernetes.io/projected/00003c75-2730-4420-86db-53ebe728b1e5-kube-api-access-85gdx\") pod \"authorino-operator-7b7bb4bc77-kqnnj\" (UID: \"00003c75-2730-4420-86db-53ebe728b1e5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:49:44.805409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:44.805371    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:49:45.203434 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:45.203398    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00003c75_2730_4420_86db_53ebe728b1e5.slice/crio-851baf664fd9625ff3760ebc5501cd3cbc069c2f10d45a7cf2f053e46ee8b34c WatchSource:0}: Error finding container 851baf664fd9625ff3760ebc5501cd3cbc069c2f10d45a7cf2f053e46ee8b34c: Status 404 returned error can't find the container with id 851baf664fd9625ff3760ebc5501cd3cbc069c2f10d45a7cf2f053e46ee8b34c
Nov 03 17:49:45.205243 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:45.205218    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"]
Nov 03 17:49:46.147948 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:46.147920    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj" event={"ID":"00003c75-2730-4420-86db-53ebe728b1e5","Type":"ContainerStarted","Data":"851baf664fd9625ff3760ebc5501cd3cbc069c2f10d45a7cf2f053e46ee8b34c"}
Nov 03 17:49:46.149777 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:46.149745    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v" event={"ID":"b9731ed5-3173-443b-8992-51c6cfc38e6f","Type":"ContainerStarted","Data":"9ce94721f8623a6dccae0653539997c02ffa741b40c9b2592551d2052a5acc9c"}
Nov 03 17:49:47.156649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:47.156612    6167 generic.go:356] "Generic (PLEG): container finished" podID="b9731ed5-3173-443b-8992-51c6cfc38e6f" containerID="9ce94721f8623a6dccae0653539997c02ffa741b40c9b2592551d2052a5acc9c" exitCode=0
Nov 03 17:49:47.157070 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:47.156659    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v" event={"ID":"b9731ed5-3173-443b-8992-51c6cfc38e6f","Type":"ContainerDied","Data":"9ce94721f8623a6dccae0653539997c02ffa741b40c9b2592551d2052a5acc9c"}
Nov 03 17:49:47.157070 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:47.156708    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v" event={"ID":"b9731ed5-3173-443b-8992-51c6cfc38e6f","Type":"ContainerStarted","Data":"1c21e8dddf7783d4bdb5ec90fde9e8ee6e8dd665fcf8b33063c81bfd5252b99e"}
Nov 03 17:49:47.157070 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:47.156792    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:49:47.314334 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:47.314225    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v" podStartSLOduration=1.537826323 podStartE2EDuration="7.31421059s" totalImagesPullingTime="5.287449022s" totalInitContainerRuntime="488.935245ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:47.312245283 +0000 UTC m=+242.542570367" watchObservedRunningTime="2026-11-03 17:49:47.31421059 +0000 UTC m=+242.544535662"
Nov 03 17:49:49.168527 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:49.168493    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj" event={"ID":"00003c75-2730-4420-86db-53ebe728b1e5","Type":"ContainerStarted","Data":"61ad096007687a88cb48213f5d11bdfbda9af06f3b0e0831cd00cdec195cf1e9"}
Nov 03 17:49:49.169899 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:49.169867    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:49:49.332512 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:49.332456    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj" podStartSLOduration=2.3135469730000002 podStartE2EDuration="5.33244422s" totalImagesPullingTime="3.018897247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:49.331255622 +0000 UTC m=+244.561580706" watchObservedRunningTime="2026-11-03 17:49:49.33244422 +0000 UTC m=+244.562769292"
Nov 03 17:49:50.117303 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:50.117273    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fss7p"
Nov 03 17:49:53.073411 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.073377    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"]
Nov 03 17:49:53.076993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.076972    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.128838 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.128811    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"]
Nov 03 17:49:53.134742 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.134708    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5xc7p"
Nov 03 17:49:53.155446 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.155417    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mlt\" (UniqueName: \"kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt\") pod \"kuadrant-operator-controller-manager-678d87766f-qhgfh\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.155552 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.155471    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qhgfh\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.177403 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.177377    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vjv2n\""
Nov 03 17:49:53.256277 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.256227    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mlt\" (UniqueName: \"kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt\") pod \"kuadrant-operator-controller-manager-678d87766f-qhgfh\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.256454 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.256309    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qhgfh\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.256672 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.256653    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qhgfh\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.318267 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.318240    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mlt\" (UniqueName: \"kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt\") pod \"kuadrant-operator-controller-manager-678d87766f-qhgfh\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.389430 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.389363    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:49:53.863496 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:49:53.863463    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34af34ed_2b37_4c4a_a3df_7d5ed334d4cd.slice/crio-79d0e5f351d80c74a4f9b5b07bb51e6845f5f7cdd5dd750eabbf510e8ced6877 WatchSource:0}: Error finding container 79d0e5f351d80c74a4f9b5b07bb51e6845f5f7cdd5dd750eabbf510e8ced6877: Status 404 returned error can't find the container with id 79d0e5f351d80c74a4f9b5b07bb51e6845f5f7cdd5dd750eabbf510e8ced6877
Nov 03 17:49:53.864912 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:53.864880    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"]
Nov 03 17:49:54.193496 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:54.193410    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh" event={"ID":"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd","Type":"ContainerStarted","Data":"79d0e5f351d80c74a4f9b5b07bb51e6845f5f7cdd5dd750eabbf510e8ced6877"}
Nov 03 17:49:54.520532 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:49:54.520449    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:49:58.165235 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:49:58.165200    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-kk49v"
Nov 03 17:50:01.181366 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:01.181332    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kqnnj"
Nov 03 17:50:01.229946 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:01.229904    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh" event={"ID":"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd","Type":"ContainerStarted","Data":"f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f"}
Nov 03 17:50:01.230122 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:01.229980    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:50:12.236518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:12.236483    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 17:50:12.459428 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:12.459375    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh" podStartSLOduration=14.10252476 podStartE2EDuration="20.459362862s" totalImagesPullingTime="6.356838102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:01.459697709 +0000 UTC m=+256.690022783" watchObservedRunningTime="2026-11-03 17:50:12.459362862 +0000 UTC m=+267.689687933"
Nov 03 17:50:19.860035 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.860006    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"]
Nov 03 17:50:19.867858 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.867831    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.924799 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.924768    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7zcwt\""
Nov 03 17:50:19.975457 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.975422    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"]
Nov 03 17:50:19.997580 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997550    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjk6m\" (UniqueName: \"kubernetes.io/projected/49d03cb8-8b24-48e0-a8a8-befc390455ac-kube-api-access-fjk6m\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997740 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997586    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997740 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997614    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997740 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997710    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997850 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997794    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997895 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997851    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997895 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997884    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d03cb8-8b24-48e0-a8a8-befc390455ac-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997958 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997941    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:19.997991 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:19.997983    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099208 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099175    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099208 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099212    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d03cb8-8b24-48e0-a8a8-befc390455ac-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099241    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099275    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099322    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjk6m\" (UniqueName: \"kubernetes.io/projected/49d03cb8-8b24-48e0-a8a8-befc390455ac-kube-api-access-fjk6m\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099350    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099392    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099417    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099444    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099653    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099864 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099777    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099923 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099904    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.099973 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.099935    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.100116 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.100096    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d03cb8-8b24-48e0-a8a8-befc390455ac-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.101862 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.101836    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.102004 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.101986    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.166021 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.165946    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d03cb8-8b24-48e0-a8a8-befc390455ac-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.166144 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.166099    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjk6m\" (UniqueName: \"kubernetes.io/projected/49d03cb8-8b24-48e0-a8a8-befc390455ac-kube-api-access-fjk6m\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs\" (UID: \"49d03cb8-8b24-48e0-a8a8-befc390455ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.182931 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.182908    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:20.620911 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.620883    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"]
Nov 03 17:50:20.621048 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:50:20.620982    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d03cb8_8b24_48e0_a8a8_befc390455ac.slice/crio-c41f3e5fbd8620aedf445debcc9e4b22b2d756ee81b004f94d65dc25cb91515e WatchSource:0}: Error finding container c41f3e5fbd8620aedf445debcc9e4b22b2d756ee81b004f94d65dc25cb91515e: Status 404 returned error can't find the container with id c41f3e5fbd8620aedf445debcc9e4b22b2d756ee81b004f94d65dc25cb91515e
Nov 03 17:50:20.623043 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.623016    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:50:20.623120 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.623081    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:50:20.623120 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:20.623112    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:50:21.324544 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:21.324510    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs" event={"ID":"49d03cb8-8b24-48e0-a8a8-befc390455ac","Type":"ContainerStarted","Data":"365ad61383b2a7d6afd8cb529c9fad50a3e4da25408a338997b9c6431497514e"}
Nov 03 17:50:21.324963 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:21.324562    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs" event={"ID":"49d03cb8-8b24-48e0-a8a8-befc390455ac","Type":"ContainerStarted","Data":"c41f3e5fbd8620aedf445debcc9e4b22b2d756ee81b004f94d65dc25cb91515e"}
Nov 03 17:50:21.510149 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:21.510091    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs" podStartSLOduration=2.510076951 podStartE2EDuration="2.510076951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:21.458448895 +0000 UTC m=+276.688773962" watchObservedRunningTime="2026-11-03 17:50:21.510076951 +0000 UTC m=+276.740402023"
Nov 03 17:50:22.183127 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:22.183089    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:22.188156 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:22.188131    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:22.329610 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:22.329580    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:22.330575 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:22.330557    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs"
Nov 03 17:50:24.520197 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:50:24.520111    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:50:25.459483 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.459453    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r2l99"]
Nov 03 17:50:25.461691 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.461676    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:50:25.468371 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.468308    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:50:25.470875 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.470856    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.521104 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.521076    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r2l99"]
Nov 03 17:50:25.526128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.526101    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 17:50:25.548510 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.548480    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6znp\" (UniqueName: \"kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp\") pod \"authorino-56c6c9ddd5-r2l99\" (UID: \"3ff80a91-7d1b-436c-897b-a9d24e2cca15\") " pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:50:25.548636 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.548563    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad065c46-6333-426a-9269-4470118668f8-config-file\") pod \"limitador-limitador-7c9b87d574-g7dk4\" (UID: \"ad065c46-6333-426a-9269-4470118668f8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.548636 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.548594    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzvw\" (UniqueName: \"kubernetes.io/projected/ad065c46-6333-426a-9269-4470118668f8-kube-api-access-sqzvw\") pod \"limitador-limitador-7c9b87d574-g7dk4\" (UID: \"ad065c46-6333-426a-9269-4470118668f8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.562344 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.562318    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:50:25.569598 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.569576    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4hst9\""
Nov 03 17:50:25.569700 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.569581    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hj9sf\""
Nov 03 17:50:25.649857 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.649826    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzvw\" (UniqueName: \"kubernetes.io/projected/ad065c46-6333-426a-9269-4470118668f8-kube-api-access-sqzvw\") pod \"limitador-limitador-7c9b87d574-g7dk4\" (UID: \"ad065c46-6333-426a-9269-4470118668f8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.650020 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.649911    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6znp\" (UniqueName: \"kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp\") pod \"authorino-56c6c9ddd5-r2l99\" (UID: \"3ff80a91-7d1b-436c-897b-a9d24e2cca15\") " pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:50:25.650020 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.649962    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad065c46-6333-426a-9269-4470118668f8-config-file\") pod \"limitador-limitador-7c9b87d574-g7dk4\" (UID: \"ad065c46-6333-426a-9269-4470118668f8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.650509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.650492    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad065c46-6333-426a-9269-4470118668f8-config-file\") pod \"limitador-limitador-7c9b87d574-g7dk4\" (UID: \"ad065c46-6333-426a-9269-4470118668f8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.712169 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.712106    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzvw\" (UniqueName: \"kubernetes.io/projected/ad065c46-6333-426a-9269-4470118668f8-kube-api-access-sqzvw\") pod \"limitador-limitador-7c9b87d574-g7dk4\" (UID: \"ad065c46-6333-426a-9269-4470118668f8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:25.712285 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.712191    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6znp\" (UniqueName: \"kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp\") pod \"authorino-56c6c9ddd5-r2l99\" (UID: \"3ff80a91-7d1b-436c-897b-a9d24e2cca15\") " pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:50:25.785108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.785080    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:50:25.786820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.786786    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:50:25.794095 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:25.794071    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:26.150867 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:50:26.150834    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ff80a91_7d1b_436c_897b_a9d24e2cca15.slice/crio-6528d00433ac7202f720e1f29880a24c23ef23b18c2ee906cd34b1f04d6f7a0c WatchSource:0}: Error finding container 6528d00433ac7202f720e1f29880a24c23ef23b18c2ee906cd34b1f04d6f7a0c: Status 404 returned error can't find the container with id 6528d00433ac7202f720e1f29880a24c23ef23b18c2ee906cd34b1f04d6f7a0c
Nov 03 17:50:26.152690 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:26.152664    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r2l99"]
Nov 03 17:50:26.202590 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:50:26.202556    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad065c46_6333_426a_9269_4470118668f8.slice/crio-0a975063c899cb4515b904ee4cd530d518716d3feafde9e58ed54ebf075f5a1f WatchSource:0}: Error finding container 0a975063c899cb4515b904ee4cd530d518716d3feafde9e58ed54ebf075f5a1f: Status 404 returned error can't find the container with id 0a975063c899cb4515b904ee4cd530d518716d3feafde9e58ed54ebf075f5a1f
Nov 03 17:50:26.203749 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:26.203707    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:50:26.348298 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:26.348262    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4" event={"ID":"ad065c46-6333-426a-9269-4470118668f8","Type":"ContainerStarted","Data":"0a975063c899cb4515b904ee4cd530d518716d3feafde9e58ed54ebf075f5a1f"}
Nov 03 17:50:26.349536 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:26.349511    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99" event={"ID":"3ff80a91-7d1b-436c-897b-a9d24e2cca15","Type":"ContainerStarted","Data":"6528d00433ac7202f720e1f29880a24c23ef23b18c2ee906cd34b1f04d6f7a0c"}
Nov 03 17:50:29.368111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:29.368067    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4" event={"ID":"ad065c46-6333-426a-9269-4470118668f8","Type":"ContainerStarted","Data":"73f7617250bd03687b45596beb65b4c40bed8d86df8349ee5f4642b384e4370e"}
Nov 03 17:50:29.368598 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:29.368187    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:29.369693 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:29.369670    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99" event={"ID":"3ff80a91-7d1b-436c-897b-a9d24e2cca15","Type":"ContainerStarted","Data":"3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7"}
Nov 03 17:50:29.492864 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:29.492744    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4" podStartSLOduration=2.150348449 podStartE2EDuration="4.492707789s" totalImagesPullingTime="2.34235934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:29.490479194 +0000 UTC m=+284.720804269" watchObservedRunningTime="2026-11-03 17:50:29.492707789 +0000 UTC m=+284.723032861"
Nov 03 17:50:29.611714 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:29.611659    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99" podStartSLOduration=1.570741272 podStartE2EDuration="4.611648956s" totalImagesPullingTime="3.040907684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:29.611311168 +0000 UTC m=+284.841636237" watchObservedRunningTime="2026-11-03 17:50:29.611648956 +0000 UTC m=+284.841974027"
Nov 03 17:50:40.376479 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:40.376447    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"
Nov 03 17:50:44.947460 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:44.947429    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:50:44.949614 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:50:44.949593    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:50:54.519954 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:50:54.519914    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:51:24.520135 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:51:24.520094    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:51:53.462382 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:53.462295    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:51:54.520284 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:51:54.520246    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:51:56.179267 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.179238    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wp8w9"]
Nov 03 17:51:56.182169 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.182152    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.268373 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.268340    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416e0881-1aac-4e94-b0a9-25e4e6083aac-cert\") pod \"kserve-controller-manager-776cc66d47-wp8w9\" (UID: \"416e0881-1aac-4e94-b0a9-25e4e6083aac\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.268544 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.268399    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftph\" (UniqueName: \"kubernetes.io/projected/416e0881-1aac-4e94-b0a9-25e4e6083aac-kube-api-access-lftph\") pod \"kserve-controller-manager-776cc66d47-wp8w9\" (UID: \"416e0881-1aac-4e94-b0a9-25e4e6083aac\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.294500 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.294469    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 17:51:56.294643 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.294512    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 17:51:56.294643 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.294521    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-k4p2f\""
Nov 03 17:51:56.294643 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.294558    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 17:51:56.296113 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.296080    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wp8w9"]
Nov 03 17:51:56.368900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.368865    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416e0881-1aac-4e94-b0a9-25e4e6083aac-cert\") pod \"kserve-controller-manager-776cc66d47-wp8w9\" (UID: \"416e0881-1aac-4e94-b0a9-25e4e6083aac\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.368900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.368905    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftph\" (UniqueName: \"kubernetes.io/projected/416e0881-1aac-4e94-b0a9-25e4e6083aac-kube-api-access-lftph\") pod \"kserve-controller-manager-776cc66d47-wp8w9\" (UID: \"416e0881-1aac-4e94-b0a9-25e4e6083aac\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.371493 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.371460    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416e0881-1aac-4e94-b0a9-25e4e6083aac-cert\") pod \"kserve-controller-manager-776cc66d47-wp8w9\" (UID: \"416e0881-1aac-4e94-b0a9-25e4e6083aac\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.455503 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.455439    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftph\" (UniqueName: \"kubernetes.io/projected/416e0881-1aac-4e94-b0a9-25e4e6083aac-kube-api-access-lftph\") pod \"kserve-controller-manager-776cc66d47-wp8w9\" (UID: \"416e0881-1aac-4e94-b0a9-25e4e6083aac\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.495386 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.495355    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:51:56.529099 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.529069    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-txqgp"]
Nov 03 17:51:56.557410 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.557375    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.570402 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.570373    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wc9\" (UniqueName: \"kubernetes.io/projected/70cb4192-1966-47b3-b571-4bc47ed43919-kube-api-access-t5wc9\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.570541 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.570418    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb4192-1966-47b3-b571-4bc47ed43919-metrics-certs\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.570541 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.570454    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb4192-1966-47b3-b571-4bc47ed43919-cert\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.590518 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.589122    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-txqgp"]
Nov 03 17:51:56.611489 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.611459    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 17:51:56.611636 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.611496    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p22n9\""
Nov 03 17:51:56.611636 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.611525    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 17:51:56.671534 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.671499    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wc9\" (UniqueName: \"kubernetes.io/projected/70cb4192-1966-47b3-b571-4bc47ed43919-kube-api-access-t5wc9\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.671693 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.671547    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb4192-1966-47b3-b571-4bc47ed43919-metrics-certs\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.671693 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.671582    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb4192-1966-47b3-b571-4bc47ed43919-cert\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.674579 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.674549    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cb4192-1966-47b3-b571-4bc47ed43919-metrics-certs\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.674710 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.674549    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb4192-1966-47b3-b571-4bc47ed43919-cert\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.756977 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.756903    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wc9\" (UniqueName: \"kubernetes.io/projected/70cb4192-1966-47b3-b571-4bc47ed43919-kube-api-access-t5wc9\") pod \"odh-model-controller-769875db56-txqgp\" (UID: \"70cb4192-1966-47b3-b571-4bc47ed43919\") " pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.874534 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.874490    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:51:56.891388 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.891363    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wp8w9"]
Nov 03 17:51:56.891950 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:51:56.891922    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416e0881_1aac_4e94_b0a9_25e4e6083aac.slice/crio-a6291c094be77d7450e5b520cbf3fe0c5a9358796a092ebb3ba3ba7a44dc3531 WatchSource:0}: Error finding container a6291c094be77d7450e5b520cbf3fe0c5a9358796a092ebb3ba3ba7a44dc3531: Status 404 returned error can't find the container with id a6291c094be77d7450e5b520cbf3fe0c5a9358796a092ebb3ba3ba7a44dc3531
Nov 03 17:51:56.893373 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:56.893349    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:51:57.258367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:57.258338    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-txqgp"]
Nov 03 17:51:57.260420 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:51:57.260392    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cb4192_1966_47b3_b571_4bc47ed43919.slice/crio-e1fe467630d9b87ede001008904a6c42be922a0d79b90f4354e0f42c9008543c WatchSource:0}: Error finding container e1fe467630d9b87ede001008904a6c42be922a0d79b90f4354e0f42c9008543c: Status 404 returned error can't find the container with id e1fe467630d9b87ede001008904a6c42be922a0d79b90f4354e0f42c9008543c
Nov 03 17:51:57.797305 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:57.797267    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-txqgp" event={"ID":"70cb4192-1966-47b3-b571-4bc47ed43919","Type":"ContainerStarted","Data":"e1fe467630d9b87ede001008904a6c42be922a0d79b90f4354e0f42c9008543c"}
Nov 03 17:51:57.798592 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:51:57.798548    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9" event={"ID":"416e0881-1aac-4e94-b0a9-25e4e6083aac","Type":"ContainerStarted","Data":"a6291c094be77d7450e5b520cbf3fe0c5a9358796a092ebb3ba3ba7a44dc3531"}
Nov 03 17:52:01.663380 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.663350    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"]
Nov 03 17:52:01.667216 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.667198    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.715763 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.715702    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtp5\" (UniqueName: \"kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.715907 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.715831    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.715907 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.715871    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.715997 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.715907    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.773611 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.773572    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:52:01.773611 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.773576    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:52:01.773855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.773634    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 17:52:01.773855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.773709    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tr54c\""
Nov 03 17:52:01.773855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.773833    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 17:52:01.775238 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.775218    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"]
Nov 03 17:52:01.817174 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.817144    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtp5\" (UniqueName: \"kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.817347 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.817187    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.817347 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.817215    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.817347 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.817243    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.818093 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.818066    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.819834 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.819786    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.820175 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.820158    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.825394 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.825334    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-txqgp" event={"ID":"70cb4192-1966-47b3-b571-4bc47ed43919","Type":"ContainerStarted","Data":"1d459e7e4141b0d6747b09e5673da536875eb29baa99535945006222e8696dbc"}
Nov 03 17:52:01.825521 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.825444    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:52:01.827290 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.827263    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9" event={"ID":"416e0881-1aac-4e94-b0a9-25e4e6083aac","Type":"ContainerStarted","Data":"c0043b7607296bd01a69a40ad3a87f110e6b6f2023b19380017e886c87acf11b"}
Nov 03 17:52:01.827404 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.827345    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:52:01.881945 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.881912    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtp5\" (UniqueName: \"kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5\") pod \"maas-api-7f85bcb458-w9878\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:01.945778 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.945669    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-txqgp" podStartSLOduration=2.000224168 podStartE2EDuration="5.945655641s" totalImagesPullingTime="3.945431473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:01.944603646 +0000 UTC m=+377.174928717" watchObservedRunningTime="2026-11-03 17:52:01.945655641 +0000 UTC m=+377.175980709"
Nov 03 17:52:01.980011 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:01.979979    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:02.065332 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.064371    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9" podStartSLOduration=1.5980074 podStartE2EDuration="6.064350697s" totalImagesPullingTime="4.466343297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:02.062624713 +0000 UTC m=+377.292949791" watchObservedRunningTime="2026-11-03 17:52:02.064350697 +0000 UTC m=+377.294675769"
Nov 03 17:52:02.401216 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.401181    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"]
Nov 03 17:52:02.402695 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:52:02.402667    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c90867b_73d5_46bf_bda6_5c71be0b9f7e.slice/crio-e8cad1cc5987dd883f43c6f0010f1b6b5b4e28e78903467dd777c21ee1f4ca09 WatchSource:0}: Error finding container e8cad1cc5987dd883f43c6f0010f1b6b5b4e28e78903467dd777c21ee1f4ca09: Status 404 returned error can't find the container with id e8cad1cc5987dd883f43c6f0010f1b6b5b4e28e78903467dd777c21ee1f4ca09
Nov 03 17:52:02.838909 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.838872    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878" event={"ID":"3c90867b-73d5-46bf-bda6-5c71be0b9f7e","Type":"ContainerStarted","Data":"e8cad1cc5987dd883f43c6f0010f1b6b5b4e28e78903467dd777c21ee1f4ca09"}
Nov 03 17:52:02.840517 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.840492    6167 generic.go:356] "Generic (PLEG): container finished" podID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerID="037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22" exitCode=0
Nov 03 17:52:02.840627 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.840562    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" event={"ID":"87b57a48-4012-4b73-9a5c-e69fde62eaf7","Type":"ContainerDied","Data":"037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"}
Nov 03 17:52:02.841038 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.841017    6167 scope.go:122] "RemoveContainer" containerID="037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"
Nov 03 17:52:02.841998 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.841972    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:52:02.842069 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.842039    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:52:02.842108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:02.842074    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:52:03.863799 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:03.863761    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" event={"ID":"87b57a48-4012-4b73-9a5c-e69fde62eaf7","Type":"ContainerStarted","Data":"da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d"}
Nov 03 17:52:04.495417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.495346    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:52:04.495417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.495402    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r597d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 17:52:04.495710 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.495454    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 17:52:04.513948 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.513910    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r2l99"]
Nov 03 17:52:04.514112 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.514083    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99" podUID="3ff80a91-7d1b-436c-897b-a9d24e2cca15" containerName="authorino" containerID="cri-o://3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7" gracePeriod=30
Nov 03 17:52:04.817420 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.817394    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:52:04.869291 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.869262    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp\" (UniqueName: \"kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp\") pod \"3ff80a91-7d1b-436c-897b-a9d24e2cca15\" (UID: \"3ff80a91-7d1b-436c-897b-a9d24e2cca15\") "
Nov 03 17:52:04.870567 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.870532    6167 generic.go:356] "Generic (PLEG): container finished" podID="3ff80a91-7d1b-436c-897b-a9d24e2cca15" containerID="3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7" exitCode=0
Nov 03 17:52:04.870758 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.870596    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99"
Nov 03 17:52:04.870758 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.870626    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99" event={"ID":"3ff80a91-7d1b-436c-897b-a9d24e2cca15","Type":"ContainerDied","Data":"3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7"}
Nov 03 17:52:04.870758 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.870665    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r2l99" event={"ID":"3ff80a91-7d1b-436c-897b-a9d24e2cca15","Type":"ContainerDied","Data":"6528d00433ac7202f720e1f29880a24c23ef23b18c2ee906cd34b1f04d6f7a0c"}
Nov 03 17:52:04.870758 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.870680    6167 scope.go:122] "RemoveContainer" containerID="3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7"
Nov 03 17:52:04.871949 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.871922    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp" pod "3ff80a91-7d1b-436c-897b-a9d24e2cca15" (UID: "3ff80a91-7d1b-436c-897b-a9d24e2cca15"). InnerVolumeSpecName "kube-api-access-k6znp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:04.883346 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.883313    6167 scope.go:122] "RemoveContainer" containerID="3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7"
Nov 03 17:52:04.883632 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:52:04.883610    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7\": container with ID starting with 3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7 not found: ID does not exist" containerID="3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7"
Nov 03 17:52:04.883682 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.883641    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7"} err="failed to get container status \"3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7\": rpc error: code = NotFound desc = could not find container \"3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7\": container with ID starting with 3fc2ac96a1e8c7941180b72c25e4668ca42f75eaf2e3cf060ab4ad9d8dadcde7 not found: ID does not exist"
Nov 03 17:52:04.970648 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:04.970195    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6znp\" (UniqueName: \"kubernetes.io/projected/3ff80a91-7d1b-436c-897b-a9d24e2cca15-kube-api-access-k6znp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:52:05.291578 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:05.291538    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r2l99"]
Nov 03 17:52:05.294132 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:05.294108    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r2l99"]
Nov 03 17:52:05.495553 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:05.495510    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r597d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 17:52:05.495779 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:05.495567    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 17:52:06.496133 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:06.496097    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r597d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 17:52:06.496567 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:06.496159    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 17:52:06.927232 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:06.927193    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff80a91-7d1b-436c-897b-a9d24e2cca15" path="/var/lib/kubelet/pods/3ff80a91-7d1b-436c-897b-a9d24e2cca15/volumes"
Nov 03 17:52:07.495571 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:07.495547    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r597d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 17:52:07.495668 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:07.495593    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 17:52:07.889460 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:07.889409    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878" event={"ID":"3c90867b-73d5-46bf-bda6-5c71be0b9f7e","Type":"ContainerStarted","Data":"f1494c3c8fa66cb01437f06179decca635b1350476eff5398bfb52f1704f7cff"}
Nov 03 17:52:07.889887 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:07.889802    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:08.018793 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:08.018715    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878" podStartSLOduration=1.9452212800000002 podStartE2EDuration="7.018701901s" totalImagesPullingTime="5.073480621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:08.017981354 +0000 UTC m=+383.248306425" watchObservedRunningTime="2026-11-03 17:52:08.018701901 +0000 UTC m=+383.249026974"
Nov 03 17:52:08.495776 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:08.495711    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r597d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 03 17:52:08.495944 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:08.495811    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 03 17:52:09.498916 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:09.498884    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:52:09.499346 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:09.499204    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:52:09.500059 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:09.500040    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 17:52:12.842873 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:12.842840    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-txqgp"
Nov 03 17:52:13.901414 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:13.901384    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 17:52:24.522310 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:52:24.522272    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:52:32.846191 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:52:32.846162    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wp8w9"
Nov 03 17:52:54.520895 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:52:54.520854    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:53:24.520170 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:53:24.520085    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:53:54.521065 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:53:54.521025    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:54:04.679640 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.679599    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"]
Nov 03 17:54:04.680276 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.680253    6167 cpu_manager.go:414] "containerMap: removing container" podUID="3ff80a91-7d1b-436c-897b-a9d24e2cca15" containerName="authorino"
Nov 03 17:54:04.680358 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.680278    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff80a91-7d1b-436c-897b-a9d24e2cca15" containerName="authorino"
Nov 03 17:54:04.680417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.680402    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff80a91-7d1b-436c-897b-a9d24e2cca15" containerName="authorino"
Nov 03 17:54:04.683609 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.683586    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.733145 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.733110    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.733315 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.733156    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.733315 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.733236    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.733397 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.733328    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.733397 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.733357    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.733474 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.733396    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgw72\" (UniqueName: \"kubernetes.io/projected/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-kube-api-access-lgw72\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.738110 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.738084    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 17:54:04.738294 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.738094    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 17:54:04.738347 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.738169    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x9qtm\""
Nov 03 17:54:04.788603 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.788568    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 17:54:04.790328 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.790301    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"]
Nov 03 17:54:04.834410 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834378    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.834590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834419    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.834590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834451    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.834590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834474    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.834590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834496    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.834590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834529    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgw72\" (UniqueName: \"kubernetes.io/projected/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-kube-api-access-lgw72\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.835005 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.834973    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.835088 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.835007    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.835088 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.835039    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.836774 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.836755    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.837030 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.837014    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.896425 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.896396    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgw72\" (UniqueName: \"kubernetes.io/projected/6f01ae27-6219-4cf1-a0b7-5e1f190c2887-kube-api-access-lgw72\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-m7nkv\" (UID: \"6f01ae27-6219-4cf1-a0b7-5e1f190c2887\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:04.917892 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.917842    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:54:04.999534 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:04.999451    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:05.376018 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:54:05.375979    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f01ae27_6219_4cf1_a0b7_5e1f190c2887.slice/crio-b9d6c16f157c8d3659a73450b41b118bdb4fb9fc159773a731dfc9495c72b9f0 WatchSource:0}: Error finding container b9d6c16f157c8d3659a73450b41b118bdb4fb9fc159773a731dfc9495c72b9f0: Status 404 returned error can't find the container with id b9d6c16f157c8d3659a73450b41b118bdb4fb9fc159773a731dfc9495c72b9f0
Nov 03 17:54:05.377588 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:05.377560    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"]
Nov 03 17:54:05.482281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:05.482238    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv" event={"ID":"6f01ae27-6219-4cf1-a0b7-5e1f190c2887","Type":"ContainerStarted","Data":"b9d6c16f157c8d3659a73450b41b118bdb4fb9fc159773a731dfc9495c72b9f0"}
Nov 03 17:54:07.497910 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.497870    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv" event={"ID":"6f01ae27-6219-4cf1-a0b7-5e1f190c2887","Type":"ContainerStarted","Data":"f22dbc376f58155936fc8dc63a8f75b2a85aeaec21eef86ebdfb4d3a97cd9dd4"}
Nov 03 17:54:07.617681 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.617623    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv" podStartSLOduration=2.061492321 podStartE2EDuration="3.617609195s" totalImagesPullingTime="1.556116874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:07.617237622 +0000 UTC m=+502.847562708" watchObservedRunningTime="2026-11-03 17:54:07.617609195 +0000 UTC m=+502.847934264"
Nov 03 17:54:07.827403 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.827366    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"]
Nov 03 17:54:07.831227 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.831204    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:07.889824 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.889793    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 17:54:07.891667 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.891640    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"]
Nov 03 17:54:07.964029 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.964000    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e691aa9-1efc-4031-b4c5-f0cfafe60438-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:07.964212 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.964110    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s22d\" (UniqueName: \"kubernetes.io/projected/8e691aa9-1efc-4031-b4c5-f0cfafe60438-kube-api-access-9s22d\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:07.964281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.964234    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:07.964344 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.964300    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:07.964399 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.964372    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:07.964449 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:07.964416    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.000457 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.000428    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:08.065866 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.065833    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s22d\" (UniqueName: \"kubernetes.io/projected/8e691aa9-1efc-4031-b4c5-f0cfafe60438-kube-api-access-9s22d\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066005 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.065883    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066005 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.065909    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066005 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.065933    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066005 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.065952    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066005 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.065976    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e691aa9-1efc-4031-b4c5-f0cfafe60438-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066394 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.066367    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066476 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.066380    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.066476 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.066422    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.068333 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.068308    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e691aa9-1efc-4031-b4c5-f0cfafe60438-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.068521 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.068504    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e691aa9-1efc-4031-b4c5-f0cfafe60438-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.127541 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.127464    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s22d\" (UniqueName: \"kubernetes.io/projected/8e691aa9-1efc-4031-b4c5-f0cfafe60438-kube-api-access-9s22d\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pd4hh\" (UID: \"8e691aa9-1efc-4031-b4c5-f0cfafe60438\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.144348 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.144321    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:08.517763 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:54:08.517730    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e691aa9_1efc_4031_b4c5_f0cfafe60438.slice/crio-91355c99ba5a76bd022d9f703d3d8c85f936bd21c9a576a50710f0c5a006ba67 WatchSource:0}: Error finding container 91355c99ba5a76bd022d9f703d3d8c85f936bd21c9a576a50710f0c5a006ba67: Status 404 returned error can't find the container with id 91355c99ba5a76bd022d9f703d3d8c85f936bd21c9a576a50710f0c5a006ba67
Nov 03 17:54:08.519511 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:08.519474    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"]
Nov 03 17:54:09.509129 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:09.509046    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh" event={"ID":"8e691aa9-1efc-4031-b4c5-f0cfafe60438","Type":"ContainerStarted","Data":"8a6aed31ae2083e97e8e1c9fb7dfa2b7afdd8f049f3ee6768cec1b86b3199cf8"}
Nov 03 17:54:09.509129 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:09.509099    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh" event={"ID":"8e691aa9-1efc-4031-b4c5-f0cfafe60438","Type":"ContainerStarted","Data":"91355c99ba5a76bd022d9f703d3d8c85f936bd21c9a576a50710f0c5a006ba67"}
Nov 03 17:54:09.629914 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:09.629851    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh" podStartSLOduration=2.020724931 podStartE2EDuration="2.62983788s" totalImagesPullingTime="609.112949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:09.628591543 +0000 UTC m=+504.858916608" watchObservedRunningTime="2026-11-03 17:54:09.62983788 +0000 UTC m=+504.860162952"
Nov 03 17:54:10.144654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:10.144621    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:15.000116 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.000086    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:15.012627 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.012599    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:15.549619 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.549589    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv"
Nov 03 17:54:15.726349 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.726318    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"]
Nov 03 17:54:15.729827 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.729806    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.786001 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.785966    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 17:54:15.788467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.788440    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"]
Nov 03 17:54:15.847591 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.847565    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.847767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.847631    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1776c1-ff08-4072-bf71-83ed8d82beb1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.847823 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.847789    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.847870 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.847832    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.847870 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.847860    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.847985 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.847969    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8q4w\" (UniqueName: \"kubernetes.io/projected/4e1776c1-ff08-4072-bf71-83ed8d82beb1-kube-api-access-r8q4w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948450 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948411    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1776c1-ff08-4072-bf71-83ed8d82beb1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948637 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948460    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948637 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948587    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948799 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948641    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948799 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948693    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8q4w\" (UniqueName: \"kubernetes.io/projected/4e1776c1-ff08-4072-bf71-83ed8d82beb1-kube-api-access-r8q4w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948905 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948882    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948956 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948929    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.948956 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.948890    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.949128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.949110    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.950807 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.950786    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1776c1-ff08-4072-bf71-83ed8d82beb1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:15.950937 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:15.950920    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1776c1-ff08-4072-bf71-83ed8d82beb1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:16.010185 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:16.010155    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8q4w\" (UniqueName: \"kubernetes.io/projected/4e1776c1-ff08-4072-bf71-83ed8d82beb1-kube-api-access-r8q4w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj\" (UID: \"4e1776c1-ff08-4072-bf71-83ed8d82beb1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:16.043051 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:16.043024    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:16.412943 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:54:16.412912    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1776c1_ff08_4072_bf71_83ed8d82beb1.slice/crio-d8007ba3fcf7e15a42a5c4944db9cab684f5946f6e88c7a9ca3888d898750b7f WatchSource:0}: Error finding container d8007ba3fcf7e15a42a5c4944db9cab684f5946f6e88c7a9ca3888d898750b7f: Status 404 returned error can't find the container with id d8007ba3fcf7e15a42a5c4944db9cab684f5946f6e88c7a9ca3888d898750b7f
Nov 03 17:54:16.414922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:16.414894    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"]
Nov 03 17:54:16.552011 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:16.551969    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj" event={"ID":"4e1776c1-ff08-4072-bf71-83ed8d82beb1","Type":"ContainerStarted","Data":"d8007ba3fcf7e15a42a5c4944db9cab684f5946f6e88c7a9ca3888d898750b7f"}
Nov 03 17:54:17.558813 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:17.558775    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj" event={"ID":"4e1776c1-ff08-4072-bf71-83ed8d82beb1","Type":"ContainerStarted","Data":"5a7c3a9ebc65da6cc7afec7e59b80b278edb4b6466abc0a5b5d10c27596eb50a"}
Nov 03 17:54:17.679054 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:17.679002    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj" podStartSLOduration=2.043692434 podStartE2EDuration="2.678986723s" totalImagesPullingTime="635.294289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:17.67747242 +0000 UTC m=+512.907797492" watchObservedRunningTime="2026-11-03 17:54:17.678986723 +0000 UTC m=+512.909311795"
Nov 03 17:54:18.043810 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:18.043775    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:18.144781 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:18.144746    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:18.157408 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:18.157384    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:18.575813 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:18.575786    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pd4hh"
Nov 03 17:54:19.447468 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:19.447433    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:54:24.521028 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:54:24.520989    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:54:26.043766 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:26.043711    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:26.056330 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:26.056305    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:26.621188 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:26.621158    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj"
Nov 03 17:54:30.036311 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.036277    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"]
Nov 03 17:54:30.040192 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.040171    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.078321 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.078283    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.078701 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.078673    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.078993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.078968    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpg8t\" (UniqueName: \"kubernetes.io/projected/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-kube-api-access-kpg8t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.080083 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.080062    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.080269 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.080251    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.080476 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.080457    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.100037 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.100007    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"]
Nov 03 17:54:30.147654 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.147622    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 17:54:30.182073 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182035    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182293 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182089    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182293 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182137    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpg8t\" (UniqueName: \"kubernetes.io/projected/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-kube-api-access-kpg8t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182293 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182166    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182293 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182198    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182293 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182262    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182693 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182666    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.182806 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.182755    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.183179 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.183154    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.185899 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.185874    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.192199 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.192174    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.245237 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.245203    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpg8t\" (UniqueName: \"kubernetes.io/projected/3fe9fc29-00d4-46f5-ab5b-df8dd964a65c-kube-api-access-kpg8t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8\" (UID: \"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.357572 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.357525    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:30.739841 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:30.739752    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"]
Nov 03 17:54:31.641235 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:31.641152    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8" event={"ID":"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c","Type":"ContainerStarted","Data":"d72ed13f03a493accfec46922c3633b4c4897a53cfc0be46c554e1e4aff10037"}
Nov 03 17:54:31.641235 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:31.641209    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8" event={"ID":"3fe9fc29-00d4-46f5-ab5b-df8dd964a65c","Type":"ContainerStarted","Data":"af69596a1cfb2cd4ecee61d4acec7ebd4a40c92adfabd7417f627f0f1390f293"}
Nov 03 17:54:31.764279 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:31.764206    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8" podStartSLOduration=2.122422229 podStartE2EDuration="2.764185897s" totalImagesPullingTime="641.763668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:31.761443904 +0000 UTC m=+526.991768987" watchObservedRunningTime="2026-11-03 17:54:31.764185897 +0000 UTC m=+526.994510970"
Nov 03 17:54:32.357807 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:32.357775    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:36.874885 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:36.874852    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:54:40.358504 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:40.358465    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:40.371249 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:40.371226    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:40.702608 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:40.702542    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8"
Nov 03 17:54:54.521343 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:54:54.521254    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:54:56.269248 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.269214    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"]
Nov 03 17:54:56.272886 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.272865    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.330770 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.330708    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"]
Nov 03 17:54:56.347107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.347082    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 17:54:56.347274 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.347130    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 17:54:56.347353 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.347293    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-7wlk2\""
Nov 03 17:54:56.441830 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.441801    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-jfzbb\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.441992 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.441926    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd7b\" (UniqueName: \"kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b\") pod \"ai-gateway-controller-5d9cd99f45-jfzbb\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.543198 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.543119    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-jfzbb\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.543359 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.543206    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd7b\" (UniqueName: \"kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b\") pod \"ai-gateway-controller-5d9cd99f45-jfzbb\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.545683 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.545655    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-jfzbb\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.605137 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.605105    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd7b\" (UniqueName: \"kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b\") pod \"ai-gateway-controller-5d9cd99f45-jfzbb\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:56.889151 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:56.889121    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:54:57.262517 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:54:57.262484    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62e4798_7d8e_4247_8a2b_b1798cf4569e.slice/crio-c1be92b8463243f1282802c2ce92f6245cdd468b090740226b7e532950bdfbe1 WatchSource:0}: Error finding container c1be92b8463243f1282802c2ce92f6245cdd468b090740226b7e532950bdfbe1: Status 404 returned error can't find the container with id c1be92b8463243f1282802c2ce92f6245cdd468b090740226b7e532950bdfbe1
Nov 03 17:54:57.264504 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:57.264474    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"]
Nov 03 17:54:57.783023 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:54:57.782986    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb" event={"ID":"c62e4798-7d8e-4247-8a2b-b1798cf4569e","Type":"ContainerStarted","Data":"c1be92b8463243f1282802c2ce92f6245cdd468b090740226b7e532950bdfbe1"}
Nov 03 17:55:04.083674 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.083640    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"]
Nov 03 17:55:04.087529 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.087513    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107337    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107512 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107369    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107512 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107387    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107512 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107407    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zp97\" (UniqueName: \"kubernetes.io/projected/bd508cab-c70c-4b66-8086-e582b8807ba6-kube-api-access-2zp97\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107674 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107551    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107674 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107615    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107757 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107677    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd508cab-c70c-4b66-8086-e582b8807ba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107757 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107750    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.107821 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.107790    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.179482 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.179448    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"]
Nov 03 17:55:04.208454 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208416    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208642 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208470    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208642 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208505    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208642 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208538    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zp97\" (UniqueName: \"kubernetes.io/projected/bd508cab-c70c-4b66-8086-e582b8807ba6-kube-api-access-2zp97\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208642 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208579    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208642 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208630    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208951 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208657    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd508cab-c70c-4b66-8086-e582b8807ba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208951 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208736    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.208951 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.208773    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.209156 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.209127    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.209242 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.209119    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.209348 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.209326    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.209388 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.209346    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.209753 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.209705    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd508cab-c70c-4b66-8086-e582b8807ba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.211110 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.211088    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.211365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.211345    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.270739 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.270695    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd508cab-c70c-4b66-8086-e582b8807ba6-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.270911 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.270814    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zp97\" (UniqueName: \"kubernetes.io/projected/bd508cab-c70c-4b66-8086-e582b8807ba6-kube-api-access-2zp97\") pod \"maas-default-gateway-openshift-default-6dcf757899-9b9xq\" (UID: \"bd508cab-c70c-4b66-8086-e582b8807ba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.401130 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.401045    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:04.821182 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:55:04.821145    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd508cab_c70c_4b66_8086_e582b8807ba6.slice/crio-ec5483dc1cb57273b7d39d0c0e70e5e0dcc4d3726deebb10e2e82ccf9375e398 WatchSource:0}: Error finding container ec5483dc1cb57273b7d39d0c0e70e5e0dcc4d3726deebb10e2e82ccf9375e398: Status 404 returned error can't find the container with id ec5483dc1cb57273b7d39d0c0e70e5e0dcc4d3726deebb10e2e82ccf9375e398
Nov 03 17:55:04.822228 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.822201    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"]
Nov 03 17:55:04.823954 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.823914    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:55:04.824053 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.823997    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:55:04.824053 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:04.824041    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:55:05.857778 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:05.857713    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" event={"ID":"bd508cab-c70c-4b66-8086-e582b8807ba6","Type":"ContainerStarted","Data":"b36c4e9b606a4a0091072a2c549ac6c2fa432d5ec200221a122485c5a01d4edd"}
Nov 03 17:55:05.858251 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:05.857797    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" event={"ID":"bd508cab-c70c-4b66-8086-e582b8807ba6","Type":"ContainerStarted","Data":"ec5483dc1cb57273b7d39d0c0e70e5e0dcc4d3726deebb10e2e82ccf9375e398"}
Nov 03 17:55:06.054763 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.052439    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podStartSLOduration=2.052418482 podStartE2EDuration="2.052418482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:06.051301047 +0000 UTC m=+561.281626121" watchObservedRunningTime="2026-11-03 17:55:06.052418482 +0000 UTC m=+561.282743555"
Nov 03 17:55:06.401407 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.401383    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:06.403155 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.403136    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:06.403228 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.403177    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:06.864792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.864684    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb" event={"ID":"c62e4798-7d8e-4247-8a2b-b1798cf4569e","Type":"ContainerStarted","Data":"c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34"}
Nov 03 17:55:06.864792 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.864774    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:55:06.982642 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:06.982589    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb" podStartSLOduration=1.934328576 podStartE2EDuration="10.982576252s" totalImagesPullingTime="9.048247676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:06.982507273 +0000 UTC m=+562.212832345" watchObservedRunningTime="2026-11-03 17:55:06.982576252 +0000 UTC m=+562.212901324"
Nov 03 17:55:07.401554 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:07.401515    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:07.401762 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:07.401589    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:08.402416 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:08.402380    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:08.402871 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:08.402435    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:09.401745 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.401694    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:09.401934 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.401773    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:09.565898 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.565864    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"]
Nov 03 17:55:09.570355 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.570335    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.587820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.587784    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7txg\" (UniqueName: \"kubernetes.io/projected/12a4aea5-1d80-48ff-acdb-73d7404cee9f-kube-api-access-v7txg\") pod \"payload-processing-55fbbbf9dc-7t5fd\" (UID: \"12a4aea5-1d80-48ff-acdb-73d7404cee9f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.587943 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.587922    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a4aea5-1d80-48ff-acdb-73d7404cee9f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7t5fd\" (UID: \"12a4aea5-1d80-48ff-acdb-73d7404cee9f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.617552 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.617523    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"]
Nov 03 17:55:09.623457 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.623431    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 17:55:09.623614 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.623569    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6v2pj\""
Nov 03 17:55:09.688787 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.688676    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a4aea5-1d80-48ff-acdb-73d7404cee9f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7t5fd\" (UID: \"12a4aea5-1d80-48ff-acdb-73d7404cee9f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.688934 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.688802    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7txg\" (UniqueName: \"kubernetes.io/projected/12a4aea5-1d80-48ff-acdb-73d7404cee9f-kube-api-access-v7txg\") pod \"payload-processing-55fbbbf9dc-7t5fd\" (UID: \"12a4aea5-1d80-48ff-acdb-73d7404cee9f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.689401 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.689381    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a4aea5-1d80-48ff-acdb-73d7404cee9f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-7t5fd\" (UID: \"12a4aea5-1d80-48ff-acdb-73d7404cee9f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.756140 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.756106    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7txg\" (UniqueName: \"kubernetes.io/projected/12a4aea5-1d80-48ff-acdb-73d7404cee9f-kube-api-access-v7txg\") pod \"payload-processing-55fbbbf9dc-7t5fd\" (UID: \"12a4aea5-1d80-48ff-acdb-73d7404cee9f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:09.884764 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:09.884734    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:10.279068 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:55:10.279030    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a4aea5_1d80_48ff_acdb_73d7404cee9f.slice/crio-60332ac88aa0dfa7679655c725e7c0347d33b5b0296a4d012b44e9b4c0fed47c WatchSource:0}: Error finding container 60332ac88aa0dfa7679655c725e7c0347d33b5b0296a4d012b44e9b4c0fed47c: Status 404 returned error can't find the container with id 60332ac88aa0dfa7679655c725e7c0347d33b5b0296a4d012b44e9b4c0fed47c
Nov 03 17:55:10.282176 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:10.282148    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"]
Nov 03 17:55:10.402102 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:10.402070    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:10.402263 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:10.402118    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:10.892622 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:10.892584    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd" event={"ID":"12a4aea5-1d80-48ff-acdb-73d7404cee9f","Type":"ContainerStarted","Data":"60332ac88aa0dfa7679655c725e7c0347d33b5b0296a4d012b44e9b4c0fed47c"}
Nov 03 17:55:11.401531 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:11.401493    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:11.401708 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:11.401594    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:12.402173 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:12.402133    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-9b9xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 17:55:12.402595 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:12.402238    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq" podUID="bd508cab-c70c-4b66-8086-e582b8807ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 17:55:13.487510 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:13.487418    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:13.488128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:13.487852    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:13.488745 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:13.488706    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-9b9xq"
Nov 03 17:55:13.912905 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:13.912851    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd" event={"ID":"12a4aea5-1d80-48ff-acdb-73d7404cee9f","Type":"ContainerStarted","Data":"1477aa336d48aa7099ac804f0e020ab6293e706acf4d1cf82202a3601c016788"}
Nov 03 17:55:13.913130 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:13.913109    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:14.032117 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:14.032058    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd" podStartSLOduration=2.156217281 podStartE2EDuration="5.032042685s" totalImagesPullingTime="2.875825404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:14.030225725 +0000 UTC m=+569.260550797" watchObservedRunningTime="2026-11-03 17:55:14.032042685 +0000 UTC m=+569.262367773"
Nov 03 17:55:17.871631 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:17.871589    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:55:24.520685 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:55:24.520637    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:55:24.928638 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:24.928612    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-7t5fd"
Nov 03 17:55:33.312220 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.312178    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"]
Nov 03 17:55:33.312888 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.312404    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb" podUID="c62e4798-7d8e-4247-8a2b-b1798cf4569e" containerName="manager" containerID="cri-o://c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34" gracePeriod=30
Nov 03 17:55:33.613621 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.613590    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:55:33.743998 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.743961    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs\") pod \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") "
Nov 03 17:55:33.744189 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.744014    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b\" (UniqueName: \"kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b\") pod \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\" (UID: \"c62e4798-7d8e-4247-8a2b-b1798cf4569e\") "
Nov 03 17:55:33.746302 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.746272    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b" pod "c62e4798-7d8e-4247-8a2b-b1798cf4569e" (UID: "c62e4798-7d8e-4247-8a2b-b1798cf4569e"). InnerVolumeSpecName "kube-api-access-bkd7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:55:33.746302 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.746277    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs" pod "c62e4798-7d8e-4247-8a2b-b1798cf4569e" (UID: "c62e4798-7d8e-4247-8a2b-b1798cf4569e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:55:33.845502 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.845427    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c62e4798-7d8e-4247-8a2b-b1798cf4569e-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:33.845502 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:33.845456    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkd7b\" (UniqueName: \"kubernetes.io/projected/c62e4798-7d8e-4247-8a2b-b1798cf4569e-kube-api-access-bkd7b\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:55:34.018731 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.018694    6167 generic.go:356] "Generic (PLEG): container finished" podID="c62e4798-7d8e-4247-8a2b-b1798cf4569e" containerID="c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34" exitCode=0
Nov 03 17:55:34.018900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.018747    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb" event={"ID":"c62e4798-7d8e-4247-8a2b-b1798cf4569e","Type":"ContainerDied","Data":"c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34"}
Nov 03 17:55:34.018900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.018762    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"
Nov 03 17:55:34.018900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.018801    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb" event={"ID":"c62e4798-7d8e-4247-8a2b-b1798cf4569e","Type":"ContainerDied","Data":"c1be92b8463243f1282802c2ce92f6245cdd468b090740226b7e532950bdfbe1"}
Nov 03 17:55:34.018900 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.018823    6167 scope.go:122] "RemoveContainer" containerID="c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34"
Nov 03 17:55:34.028637 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.028621    6167 scope.go:122] "RemoveContainer" containerID="c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34"
Nov 03 17:55:34.029009 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:55:34.028988    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34\": container with ID starting with c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34 not found: ID does not exist" containerID="c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34"
Nov 03 17:55:34.029084 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.029022    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34"} err="failed to get container status \"c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34\": rpc error: code = NotFound desc = could not find container \"c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34\": container with ID starting with c85fdfd2363fe8e0f5dbed6e003650f9e6adb5dcad00e2192ea5e44b208d1e34 not found: ID does not exist"
Nov 03 17:55:34.185645 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.185565    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"]
Nov 03 17:55:34.236361 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.236328    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-jfzbb"]
Nov 03 17:55:34.932640 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:34.932592    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62e4798-7d8e-4247-8a2b-b1798cf4569e" path="/var/lib/kubelet/pods/c62e4798-7d8e-4247-8a2b-b1798cf4569e/volumes"
Nov 03 17:55:44.991950 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:44.991919    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:55:44.992668 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:55:44.992642    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 17:55:54.520598 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:55:54.520562    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:56:07.553968 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.553936    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"]
Nov 03 17:56:07.554440 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.554426    6167 cpu_manager.go:414] "containerMap: removing container" podUID="c62e4798-7d8e-4247-8a2b-b1798cf4569e" containerName="manager"
Nov 03 17:56:07.554488 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.554441    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62e4798-7d8e-4247-8a2b-b1798cf4569e" containerName="manager"
Nov 03 17:56:07.554545 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.554536    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62e4798-7d8e-4247-8a2b-b1798cf4569e" containerName="manager"
Nov 03 17:56:07.557810 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.557791    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.611381 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.611352    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2b4f46-openshift-default-dockercfg-5qdjt\""
Nov 03 17:56:07.663224 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.663190    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"]
Nov 03 17:56:07.676542 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676511    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.676701 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676551    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.676701 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676649    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.676860 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676744    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.676860 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676778    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.676860 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676819    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.677010 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676863    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.677010 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676892    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24hx\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.677010 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.676937    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778001 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.777962    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24hx\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778001 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778001    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778057    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778080    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778098    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778145    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778167    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778244 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778193    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778571 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778251    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778571 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778538    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778678 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778566    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778801 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778780    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.778801 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.778790    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.779125 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.779098    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.780850 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.780823    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.780850 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.780839    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.848119 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.848089    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24hx\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.849699 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.849676    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.870624 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.870598    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"]
Nov 03 17:56:07.870750 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.870603    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:07.875509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.875328    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.930277 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.930245    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aab753-openshift-default-dockercfg-2fhq6\""
Nov 03 17:56:07.946633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.945474    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"]
Nov 03 17:56:07.980647 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980614    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.980767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980688    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.980767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980749    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.980848 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980778    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.980848 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980822    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.980915 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980866    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.980948 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.980910    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.981256 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.981019    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:07.981256 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:07.981049    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddddx\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.082680 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082642    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.082883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082697    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddddx\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.082883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082761    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.082883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082831    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.082883 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082877    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082900    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082947    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.082992    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.083042    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083533 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.083503    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.083543    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.083587    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083805 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.083694    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.083915 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.083892    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.085849 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.085823    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.085965 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.085920    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.144550 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.144475    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.144699 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.144614    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddddx\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.204174 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.204144    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"]
Nov 03 17:56:08.208979 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.208960    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.232687 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.216292    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:08.262480 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.262403    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0ccbb9-openshift-default-dockercfg-8dgdl\""
Nov 03 17:56:08.273661 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.269021    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"]
Nov 03 17:56:08.286072 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286045    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286165 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286086    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2t9\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286165 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286116    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286165 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286141    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286265 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286186    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286265 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286245    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286339 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286320    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286372 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286349    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.286475 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.286457    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.312368 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:08.312336    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251f15e8_9939_48fc_b99d_7ac8ab7cdedf.slice/crio-8b58b9da1b9a8de354313ad2d316b8ff2e68cc0ab37940e0dbe2d9409a13c5d0 WatchSource:0}: Error finding container 8b58b9da1b9a8de354313ad2d316b8ff2e68cc0ab37940e0dbe2d9409a13c5d0: Status 404 returned error can't find the container with id 8b58b9da1b9a8de354313ad2d316b8ff2e68cc0ab37940e0dbe2d9409a13c5d0
Nov 03 17:56:08.314901 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.314864    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.314996 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.314935    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.314996 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.314947    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"]
Nov 03 17:56:08.314996 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.314971    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.387245 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387217    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387383 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387258    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387383 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387349    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387557 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387387    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387557 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387418    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2t9\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387557 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387450    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387557 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387477    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387557 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387535    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.387771 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.387619    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.388288 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.388139    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.388288 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.388230    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.388444 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.388321    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.388688 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.388631    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.389152 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.389125    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.390105 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.390079    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.390434 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.390410    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.465623 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.465550    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2t9\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.466545 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.466526    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.536141 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.536110    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:08.660063 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.660021    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.660542 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.660124    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.660542 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.660170    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.662365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.661398    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"]
Nov 03 17:56:08.927456 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:08.927402    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21faf221_c045_4d19_88ca_6a2fd72667be.slice/crio-d1c6df7f891950ce28738da1442dbfef293e4cfebad71c169424ce325d92fc77 WatchSource:0}: Error finding container d1c6df7f891950ce28738da1442dbfef293e4cfebad71c169424ce325d92fc77: Status 404 returned error can't find the container with id d1c6df7f891950ce28738da1442dbfef293e4cfebad71c169424ce325d92fc77
Nov 03 17:56:08.929135 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.929112    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"]
Nov 03 17:56:08.929590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.929554    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.929677 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.929633    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:08.929810 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:08.929677    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:09.205467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.204879    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" event={"ID":"38d3bbfc-f466-49d3-b876-910aa65efd6a","Type":"ContainerStarted","Data":"863a366185789d3e8b6a4030e0c28334a7240f10ce8c4509be8297c524891c1d"}
Nov 03 17:56:09.205467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.204940    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" event={"ID":"38d3bbfc-f466-49d3-b876-910aa65efd6a","Type":"ContainerStarted","Data":"4c61e77a3caf7546d285aa5b582f13bc4a18f123eb544740d589d1ede1d41df0"}
Nov 03 17:56:09.206785 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.206747    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" event={"ID":"21faf221-c045-4d19-88ca-6a2fd72667be","Type":"ContainerStarted","Data":"d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"}
Nov 03 17:56:09.206785 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.206811    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" event={"ID":"21faf221-c045-4d19-88ca-6a2fd72667be","Type":"ContainerStarted","Data":"d1c6df7f891950ce28738da1442dbfef293e4cfebad71c169424ce325d92fc77"}
Nov 03 17:56:09.208590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.208558    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" event={"ID":"251f15e8-9939-48fc-b99d-7ac8ab7cdedf","Type":"ContainerStarted","Data":"4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"}
Nov 03 17:56:09.208697 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.208607    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" event={"ID":"251f15e8-9939-48fc-b99d-7ac8ab7cdedf","Type":"ContainerStarted","Data":"8b58b9da1b9a8de354313ad2d316b8ff2e68cc0ab37940e0dbe2d9409a13c5d0"}
Nov 03 17:56:09.216899 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.216878    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:09.222545 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.222526    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:09.329409 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.329359    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podStartSLOduration=2.329344646 podStartE2EDuration="2.329344646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:09.327241456 +0000 UTC m=+624.557566527" watchObservedRunningTime="2026-11-03 17:56:09.329344646 +0000 UTC m=+624.559669718"
Nov 03 17:56:09.536285 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.536251    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:09.573633 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.573567    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podStartSLOduration=2.573550657 podStartE2EDuration="2.573550657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:09.572286577 +0000 UTC m=+624.802611651" watchObservedRunningTime="2026-11-03 17:56:09.573550657 +0000 UTC m=+624.803875729"
Nov 03 17:56:09.699514 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.699410    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podStartSLOduration=1.699392851 podStartE2EDuration="1.699392851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:09.697785863 +0000 UTC m=+624.928110938" watchObservedRunningTime="2026-11-03 17:56:09.699392851 +0000 UTC m=+624.929717922"
Nov 03 17:56:09.871459 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.871420    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:09.876897 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:09.876871    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:10.213555 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:10.213520    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:10.213760 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:10.213574    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:10.214397 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:10.214367    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:10.214497 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:10.214480    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:56:10.541464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:10.541389    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:11.218539 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:11.218508    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:11.219465 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:11.219442    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:56:17.442022 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.441990    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"]
Nov 03 17:56:17.446804 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.446787    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.505862 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.505834    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"]
Nov 03 17:56:17.554015 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.553987    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-ef9633-serving-cert\""
Nov 03 17:56:17.584810 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.584776    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.584997 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.584834    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.584997 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.584968    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.585110 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.585095    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.685917 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.685884    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.686126 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.685953    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.686126 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.685987    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.686126 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.686024    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.686308 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:17.686178    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-serving-cert: secret "maas-api-e2e-worker-w1-ef9633-serving-cert" not found
Nov 03 17:56:17.686308 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:17.686277    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls podName:eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad nodeName:}" failed. No retries permitted until 2026-11-03 17:56:18.186251871 +0000 UTC m=+633.416576944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls") pod "maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" (UID: "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad") : secret "maas-api-e2e-worker-w1-ef9633-serving-cert" not found
Nov 03 17:56:17.686758 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.686730    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.688546 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.688524    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:17.753677 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:17.753599    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:18.190493 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:18.190456    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:18.192975 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:18.192948    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:18.361467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:18.361429    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:18.731142 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:18.731087    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae8c4b2_dfe0_4c5f_8618_aa2612fa2dad.slice/crio-cf431d14a3c8adca859029b39606cc5e71fed870fff3a4ad1439d9e194d734a8 WatchSource:0}: Error finding container cf431d14a3c8adca859029b39606cc5e71fed870fff3a4ad1439d9e194d734a8: Status 404 returned error can't find the container with id cf431d14a3c8adca859029b39606cc5e71fed870fff3a4ad1439d9e194d734a8
Nov 03 17:56:18.732925 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:18.732898    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"]
Nov 03 17:56:19.264219 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:19.264188    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" event={"ID":"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad","Type":"ContainerStarted","Data":"cf431d14a3c8adca859029b39606cc5e71fed870fff3a4ad1439d9e194d734a8"}
Nov 03 17:56:20.271941 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:20.271901    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" event={"ID":"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad","Type":"ContainerStarted","Data":"169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3"}
Nov 03 17:56:20.272332 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:20.272100    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:20.392386 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:20.392332    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" podStartSLOduration=2.518390783 podStartE2EDuration="3.392318526s" totalImagesPullingTime="873.927743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:20.390595661 +0000 UTC m=+635.620920732" watchObservedRunningTime="2026-11-03 17:56:20.392318526 +0000 UTC m=+635.622643594"
Nov 03 17:56:22.047498 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.047464    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"]
Nov 03 17:56:22.051601 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.051578    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.104790 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.104762    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ef9633\""
Nov 03 17:56:22.155331 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.155299    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ef9633-dockercfg-dcghn\""
Nov 03 17:56:22.157402 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.157381    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"]
Nov 03 17:56:22.239504 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.239475    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltskr\" (UniqueName: \"kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr\") pod \"payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.239669 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.239518    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.340185 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.340154    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltskr\" (UniqueName: \"kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr\") pod \"payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.340358 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.340204    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.340961 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.340941    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.402275 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.402243    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltskr\" (UniqueName: \"kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr\") pod \"payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.664997 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.664914    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:22.688399 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.688371    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"]
Nov 03 17:56:22.692857 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.692828    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.745342 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.745269    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tjp\" (UniqueName: \"kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.749136 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.748749    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.749136 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.749130    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.749732 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.749696    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.754924 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.754863    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"]
Nov 03 17:56:22.800941 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.800911    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-aab753-serving-cert\""
Nov 03 17:56:22.850828 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.850797    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.851023 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.850850    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tjp\" (UniqueName: \"kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.851023 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.850890    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.851023 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.850969    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.851195 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:22.851093    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-serving-cert: secret "maas-api-e2e-worker-w6-aab753-serving-cert" not found
Nov 03 17:56:22.851195 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:22.851162    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls podName:207b02f4-b43d-4c8a-9dd8-fd77189406dc nodeName:}" failed. No retries permitted until 2026-11-03 17:56:23.351143552 +0000 UTC m=+638.581468628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls") pod "maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" (UID: "207b02f4-b43d-4c8a-9dd8-fd77189406dc") : secret "maas-api-e2e-worker-w6-aab753-serving-cert" not found
Nov 03 17:56:22.851555 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.851531    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.853644 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.853611    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:22.916891 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:22.916820    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tjp\" (UniqueName: \"kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:23.049325 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:23.049293    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0987cbc1_cbe0_4eda_b554_82f2dc76e4cd.slice/crio-abc47ad9c24400baf565512b24d10921bdd061e6b6957ee9e96073be6a514bcd WatchSource:0}: Error finding container abc47ad9c24400baf565512b24d10921bdd061e6b6957ee9e96073be6a514bcd: Status 404 returned error can't find the container with id abc47ad9c24400baf565512b24d10921bdd061e6b6957ee9e96073be6a514bcd
Nov 03 17:56:23.052077 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.052011    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"]
Nov 03 17:56:23.291146 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.291048    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" event={"ID":"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd","Type":"ContainerStarted","Data":"494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd"}
Nov 03 17:56:23.291146 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.291109    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" event={"ID":"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd","Type":"ContainerStarted","Data":"abc47ad9c24400baf565512b24d10921bdd061e6b6957ee9e96073be6a514bcd"}
Nov 03 17:56:23.291379 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.291252    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:23.356587 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.356553    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:23.359011 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.358991    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\") pod \"maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:23.408382 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.408334    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" podStartSLOduration=2.408320191 podStartE2EDuration="2.408320191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:23.407555778 +0000 UTC m=+638.637880849" watchObservedRunningTime="2026-11-03 17:56:23.408320191 +0000 UTC m=+638.638645259"
Nov 03 17:56:23.610639 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:23.610607    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:24.033090 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:24.033061    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207b02f4_b43d_4c8a_9dd8_fd77189406dc.slice/crio-525ed2588d636152493089a0a31edd9439e34b8a4550782b09fba92a0df73755 WatchSource:0}: Error finding container 525ed2588d636152493089a0a31edd9439e34b8a4550782b09fba92a0df73755: Status 404 returned error can't find the container with id 525ed2588d636152493089a0a31edd9439e34b8a4550782b09fba92a0df73755
Nov 03 17:56:24.035928 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:24.035905    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"]
Nov 03 17:56:24.300410 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:24.300307    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" event={"ID":"207b02f4-b43d-4c8a-9dd8-fd77189406dc","Type":"ContainerStarted","Data":"525ed2588d636152493089a0a31edd9439e34b8a4550782b09fba92a0df73755"}
Nov 03 17:56:24.522347 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:24.522309    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:56:25.307767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:25.307648    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" event={"ID":"207b02f4-b43d-4c8a-9dd8-fd77189406dc","Type":"ContainerStarted","Data":"e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b"}
Nov 03 17:56:25.307767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:25.307750    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:25.428702 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:25.428644    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" podStartSLOduration=2.531158653 podStartE2EDuration="3.428631674s" totalImagesPullingTime="897.473021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:25.426878131 +0000 UTC m=+640.657203202" watchObservedRunningTime="2026-11-03 17:56:25.428631674 +0000 UTC m=+640.658956748"
Nov 03 17:56:26.104078 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:26.104043    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:56:26.283523 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:26.283499    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 17:56:27.730047 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.730013    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"]
Nov 03 17:56:27.734329 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.734309    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.788943 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.788918    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-a86228-serving-cert\""
Nov 03 17:56:27.817367 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.817333    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jztk\" (UniqueName: \"kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.817533 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.817399    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.817533 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.817487    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.817673 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.817632    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.841599 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.841569    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"]
Nov 03 17:56:27.919199 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.919162    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.919365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.919213    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jztk\" (UniqueName: \"kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.919365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.919263    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.919365 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.919279    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.919365 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:27.919347    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-serving-cert: secret "maas-api-e2e-worker-w4-a86228-serving-cert" not found
Nov 03 17:56:27.919504 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:27.919425    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls podName:b2db6fdf-7669-4ed8-9299-4100c1850311 nodeName:}" failed. No retries permitted until 2026-11-03 17:56:28.419404047 +0000 UTC m=+643.649729098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls") pod "maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" (UID: "b2db6fdf-7669-4ed8-9299-4100c1850311") : secret "maas-api-e2e-worker-w4-a86228-serving-cert" not found
Nov 03 17:56:27.920026 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.919993    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.921904 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.921880    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:27.981967 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:27.981883    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jztk\" (UniqueName: \"kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:28.424797 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:28.424760    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:28.427209 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:28.427180    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:28.647286 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:28.647233    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:29.023417 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:29.023369    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2db6fdf_7669_4ed8_9299_4100c1850311.slice/crio-2750d0ab129bc4dabc47bda3ca34da88b514b4d5866f5fb6a902d30ec2a288f5 WatchSource:0}: Error finding container 2750d0ab129bc4dabc47bda3ca34da88b514b4d5866f5fb6a902d30ec2a288f5: Status 404 returned error can't find the container with id 2750d0ab129bc4dabc47bda3ca34da88b514b4d5866f5fb6a902d30ec2a288f5
Nov 03 17:56:29.026148 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:29.026123    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"]
Nov 03 17:56:29.330312 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:29.330267    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" event={"ID":"b2db6fdf-7669-4ed8-9299-4100c1850311","Type":"ContainerStarted","Data":"2750d0ab129bc4dabc47bda3ca34da88b514b4d5866f5fb6a902d30ec2a288f5"}
Nov 03 17:56:30.336740 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:30.336688    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" event={"ID":"b2db6fdf-7669-4ed8-9299-4100c1850311","Type":"ContainerStarted","Data":"af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996"}
Nov 03 17:56:30.337097 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:30.336769    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:30.456424 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:30.456363    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" podStartSLOduration=2.651464616 podStartE2EDuration="3.456345314s" totalImagesPullingTime="804.880698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:30.45602865 +0000 UTC m=+645.686353722" watchObservedRunningTime="2026-11-03 17:56:30.456345314 +0000 UTC m=+645.686670389"
Nov 03 17:56:31.318990 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:31.318960    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 17:56:34.304179 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:34.304149    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 17:56:35.056248 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:35.056214    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:56:36.347331 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:36.347297    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 17:56:42.290767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.290735    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"]
Nov 03 17:56:42.296250 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.296232    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.352537 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.352498    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"]
Nov 03 17:56:42.383691 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.383655    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.383906 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.383733    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twn8m\" (UniqueName: \"kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m\") pod \"payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.398690 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.398664    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a86228\""
Nov 03 17:56:42.398690 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.398684    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a86228-dockercfg-54jbg\""
Nov 03 17:56:42.485111 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.485076    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twn8m\" (UniqueName: \"kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m\") pod \"payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.485303 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.485264    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.486265 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.486238    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.548262 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.548176    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twn8m\" (UniqueName: \"kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m\") pod \"payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.610252 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.610220    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:42.992192 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:42.992161    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad2dad4_78e8_4a6f_81f9_1c004379ee3a.slice/crio-8b24276c507c0f06ecaa97f9a892adb262ffca748c24cf29f300d9643e4e7c23 WatchSource:0}: Error finding container 8b24276c507c0f06ecaa97f9a892adb262ffca748c24cf29f300d9643e4e7c23: Status 404 returned error can't find the container with id 8b24276c507c0f06ecaa97f9a892adb262ffca748c24cf29f300d9643e4e7c23
Nov 03 17:56:42.995712 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:42.995681    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"]
Nov 03 17:56:43.415198 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:43.415162    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" event={"ID":"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a","Type":"ContainerStarted","Data":"a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5"}
Nov 03 17:56:43.415595 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:43.415218    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" event={"ID":"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a","Type":"ContainerStarted","Data":"8b24276c507c0f06ecaa97f9a892adb262ffca748c24cf29f300d9643e4e7c23"}
Nov 03 17:56:43.415595 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:43.415353    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:43.533230 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:43.533167    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" podStartSLOduration=1.533146371 podStartE2EDuration="1.533146371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:43.531997294 +0000 UTC m=+658.762322366" watchObservedRunningTime="2026-11-03 17:56:43.533146371 +0000 UTC m=+658.763471435"
Nov 03 17:56:43.609981 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:43.609952    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:56:47.243744 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.229788    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"]
Nov 03 17:56:47.250281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.250254    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.291732 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.291688    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"]
Nov 03 17:56:47.344322 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.344291    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f79c02-serving-cert\""
Nov 03 17:56:47.368511 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.368475    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.368704 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.368549    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.368704 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.368689    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w249t\" (UniqueName: \"kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.369025 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.368999    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.469713 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.469678    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.469922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.469752    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.469922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.469814    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.469922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.469855    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w249t\" (UniqueName: \"kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.470534 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.470505    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.472221 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.472200    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.472402 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.472380    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.556864 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.556783    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w249t\" (UniqueName: \"kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t\") pod \"maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.568843 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.568812    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:47.973264 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:56:47.973228    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab055d1_5a28_41cf_aea7_e1fb050e8ee4.slice/crio-6cbc098206d067fadbeae85ccfaeb46e92691db282a06b3091f58289139ec4b7 WatchSource:0}: Error finding container 6cbc098206d067fadbeae85ccfaeb46e92691db282a06b3091f58289139ec4b7: Status 404 returned error can't find the container with id 6cbc098206d067fadbeae85ccfaeb46e92691db282a06b3091f58289139ec4b7
Nov 03 17:56:47.975704 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:47.975683    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"]
Nov 03 17:56:48.476281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:48.476247    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" event={"ID":"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4","Type":"ContainerStarted","Data":"6cbc098206d067fadbeae85ccfaeb46e92691db282a06b3091f58289139ec4b7"}
Nov 03 17:56:49.484498 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:49.484458    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" event={"ID":"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4","Type":"ContainerStarted","Data":"0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c"}
Nov 03 17:56:49.484938 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:49.484592    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:49.612413 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:49.612350    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" podStartSLOduration=1.7666683060000001 podStartE2EDuration="2.612336289s" totalImagesPullingTime="845.667983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:49.611757233 +0000 UTC m=+664.842082302" watchObservedRunningTime="2026-11-03 17:56:49.612336289 +0000 UTC m=+664.842661354"
Nov 03 17:56:54.424298 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.424270    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 17:56:54.517023 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.516990    6167 generic.go:356] "Generic (PLEG): container finished" podID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerID="4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f" exitCode=0
Nov 03 17:56:54.517183 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.517059    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" event={"ID":"251f15e8-9939-48fc-b99d-7ac8ab7cdedf","Type":"ContainerDied","Data":"4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"}
Nov 03 17:56:54.517603 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.517476    6167 scope.go:122] "RemoveContainer" containerID="4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"
Nov 03 17:56:54.519024 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.518983    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:54.519159 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.519080    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:54.519159 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:54.519127    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:56:54.521793 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:56:54.521766    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:56:55.214104 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:55.214012    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:55.496212 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:55.496134    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 17:56:55.525274 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:55.525241    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" event={"ID":"251f15e8-9939-48fc-b99d-7ac8ab7cdedf","Type":"ContainerStarted","Data":"e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc"}
Nov 03 17:56:55.871392 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:55.871351    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:56:55.872307 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:55.872284    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 17:56:55.872417 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:55.872352    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 17:56:56.871486 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:56.871446    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 17:56:56.871912 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:56.871515    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 17:56:57.259993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:57.259904    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:56:57.259993 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:57.259973    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:56:57.871770 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:57.871704    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 17:56:57.872175 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:57.871805    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 17:56:58.871640 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:58.871600    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 17:56:58.871869 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:58.871683    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 17:56:59.645743 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:59.643944    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:56:59.871228 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:59.871192    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 17:56:59.871399 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:56:59.871250    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 17:57:00.579092 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:00.579059    6167 generic.go:356] "Generic (PLEG): container finished" podID="21faf221-c045-4d19-88ca-6a2fd72667be" containerID="d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a" exitCode=0
Nov 03 17:57:00.579521 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:00.579137    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" event={"ID":"21faf221-c045-4d19-88ca-6a2fd72667be","Type":"ContainerDied","Data":"d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"}
Nov 03 17:57:00.579607 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:00.579589    6167 scope.go:122] "RemoveContainer" containerID="d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"
Nov 03 17:57:00.581029 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:00.581001    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:57:00.581122 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:00.581070    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:57:00.581122 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:00.581103    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:57:01.018539 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.018455    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:57:01.018909 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.018888    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:57:01.019983 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.019961    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 17:57:01.086492 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.086442    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"]
Nov 03 17:57:01.090704 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.090681    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.144905 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.144871    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-0adcbc-serving-cert\""
Nov 03 17:57:01.166629 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.166592    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"]
Nov 03 17:57:01.185820 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.185782    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.195595 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.195560    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.195966 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.195943    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cw5q\" (UniqueName: \"kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.196157 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.196140    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.298410 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.298310    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.299253 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.299229    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.299446 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.299429    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cw5q\" (UniqueName: \"kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.299583 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.299570    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.299822 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:57:01.298688    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-serving-cert: secret "maas-api-e2e-worker-w3-0adcbc-serving-cert" not found
Nov 03 17:57:01.300026 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:57:01.300011    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls podName:89064973-1a13-4dca-80b5-6e9ee9c1d9fa nodeName:}" failed. No retries permitted until 2026-11-03 17:57:01.799988309 +0000 UTC m=+677.030313376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls") pod "maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" (UID: "89064973-1a13-4dca-80b5-6e9ee9c1d9fa") : secret "maas-api-e2e-worker-w3-0adcbc-serving-cert" not found
Nov 03 17:57:01.300455 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.300432    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.303383 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.303362    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.388732 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.388682    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cw5q\" (UniqueName: \"kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.591889 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.591853    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" event={"ID":"21faf221-c045-4d19-88ca-6a2fd72667be","Type":"ContainerStarted","Data":"db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf"}
Nov 03 17:57:01.806849 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.806812    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:01.809305 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:01.809275    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:02.004488 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.004408    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:02.449743 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.446380    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"]
Nov 03 17:57:02.490984 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.489510    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:57:02.541754 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.538130    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:57:02.541754 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.538253    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 17:57:02.541754 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.538297    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 17:57:02.608660 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:02.608612    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" event={"ID":"89064973-1a13-4dca-80b5-6e9ee9c1d9fa","Type":"ContainerStarted","Data":"656525aa77f961356bbf6f2181d9de88de9349efce56167cccb8144ac1a275f3"}
Nov 03 17:57:03.550584 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:03.537146    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 17:57:03.550584 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:03.537206    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 17:57:03.616673 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:03.616628    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" event={"ID":"89064973-1a13-4dca-80b5-6e9ee9c1d9fa","Type":"ContainerStarted","Data":"f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81"}
Nov 03 17:57:03.617117 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:03.616871    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:04.539183 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:04.539143    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 17:57:04.539466 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:04.539435    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 17:57:05.542886 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:05.542848    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 17:57:05.543472 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:05.543439    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 17:57:06.538742 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:06.538122    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 17:57:06.538742 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:06.538188    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 17:57:07.541560 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:07.541527    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:57:07.542056 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:07.541967    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:57:07.542948 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:07.542930    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 17:57:07.767772 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:07.767681    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" podStartSLOduration=5.85852339 podStartE2EDuration="6.767661738s" totalImagesPullingTime="909.138348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:03.768995826 +0000 UTC m=+678.999320901" watchObservedRunningTime="2026-11-03 17:57:07.767661738 +0000 UTC m=+682.997986812"
Nov 03 17:57:09.631467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:09.631435    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 17:57:10.215281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.215187    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:57:10.215281 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.215246    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 17:57:10.703017 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.702983    6167 generic.go:356] "Generic (PLEG): container finished" podID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerID="863a366185789d3e8b6a4030e0c28334a7240f10ce8c4509be8297c524891c1d" exitCode=0
Nov 03 17:57:10.703430 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.703052    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" event={"ID":"38d3bbfc-f466-49d3-b876-910aa65efd6a","Type":"ContainerDied","Data":"863a366185789d3e8b6a4030e0c28334a7240f10ce8c4509be8297c524891c1d"}
Nov 03 17:57:10.703480 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.703457    6167 scope.go:122] "RemoveContainer" containerID="863a366185789d3e8b6a4030e0c28334a7240f10ce8c4509be8297c524891c1d"
Nov 03 17:57:10.705043 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.705011    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:57:10.705157 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.705078    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:57:10.705157 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:10.705110    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 17:57:11.711517 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:11.711480    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" event={"ID":"38d3bbfc-f466-49d3-b876-910aa65efd6a","Type":"ContainerStarted","Data":"4a67a8e20f39b7f93dd251b412ebcd1f6ca98c21ec69f23aae06d091754c1b1b"}
Nov 03 17:57:12.217502 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:12.217471    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:57:12.217691 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:12.217671    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.217776 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:12.217754    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 17:57:13.217501 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:13.217463    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.217922 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:13.217534    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 17:57:14.217987 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:14.217954    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.218455 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:14.218006    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 17:57:15.217297 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:15.217261    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.217487 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:15.217320    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 17:57:16.217206 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:16.217168    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.217596 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:16.217231    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 17:57:17.221729 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:17.221683    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:57:17.222223 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:17.222090    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:57:17.222662 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:17.222641    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 17:57:19.081200 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:19.081165    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:57:24.524327 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:57:24.524289    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:57:45.757148 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:45.757113    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:57:52.195469 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.195430    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"]
Nov 03 17:57:52.200578 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.200561    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.252014 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.251981    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"]
Nov 03 17:57:52.254217 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.254188    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"simulator-w0-0ccbb9-kserve-self-signed-certs\""
Nov 03 17:57:52.254380 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.254351    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"default-dockercfg-bw54c\""
Nov 03 17:57:52.323209 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.323183    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"kube-root-ca.crt\""
Nov 03 17:57:52.323361 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.323235    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"openshift-service-ca.crt\""
Nov 03 17:57:52.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.330115    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.330231    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.330292    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.330356    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hbf\" (UniqueName: \"kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.330480    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.330519    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432287 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432251    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432333    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hbf\" (UniqueName: \"kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432385    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432464 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432425    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432627 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432549    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432627 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432584    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432767 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432700    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.432970 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432943    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.433077 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.432978    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.434783 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.434752    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.435213 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.435193    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.494361 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.494288    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hbf\" (UniqueName: \"kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf\") pod \"simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") " pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.529570 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.529537    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:57:52.924113 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:57:52.924067    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd98f3a96_0e02_4582_a7c3_574cbb2aed61.slice/crio-c3b69673697b8a7753e18bce00385fe8d71f04faacf16c5c33d3b192e4a80511 WatchSource:0}: Error finding container c3b69673697b8a7753e18bce00385fe8d71f04faacf16c5c33d3b192e4a80511: Status 404 returned error can't find the container with id c3b69673697b8a7753e18bce00385fe8d71f04faacf16c5c33d3b192e4a80511
Nov 03 17:57:52.928380 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.928359    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"]
Nov 03 17:57:52.985989 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:52.985957    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" event={"ID":"d98f3a96-0e02-4582-a7c3-574cbb2aed61","Type":"ContainerStarted","Data":"c3b69673697b8a7753e18bce00385fe8d71f04faacf16c5c33d3b192e4a80511"}
Nov 03 17:57:53.998947 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:53.998906    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" event={"ID":"d98f3a96-0e02-4582-a7c3-574cbb2aed61","Type":"ContainerStarted","Data":"9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651"}
Nov 03 17:57:54.140797 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:54.140712    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podStartSLOduration=2.140691277 podStartE2EDuration="2.140691277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:54.127425284 +0000 UTC m=+729.357750357" watchObservedRunningTime="2026-11-03 17:57:54.140691277 +0000 UTC m=+729.371016350"
Nov 03 17:57:54.521417 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:57:54.521374    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:57:54.530590 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:57:54.530572    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:58:01.735706 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:01.735672    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:58:02.530166 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:02.530125    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:58:02.550315 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:02.550282    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:58:03.065064 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:03.065034    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 17:58:20.508153 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.508119    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"]
Nov 03 17:58:20.513306 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.513285    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.568540 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.568509    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"]
Nov 03 17:58:20.616503 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.616465    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"premium-w1-ef9633-kserve-self-signed-certs\""
Nov 03 17:58:20.616659 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.616513    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"default-dockercfg-6vdlf\""
Nov 03 17:58:20.617419 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.617402    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"kube-root-ca.crt\""
Nov 03 17:58:20.617501 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.617418    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"openshift-service-ca.crt\""
Nov 03 17:58:20.655120 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.655087    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.655593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.655160    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdjc\" (UniqueName: \"kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.655593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.655226    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.655593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.655251    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.655593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.655288    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.655593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.655375    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.756674 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.756638    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.756910 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.756772    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.756910 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.756812    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdjc\" (UniqueName: \"kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.756910 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.756835    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.757103 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.756942    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.757103 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.756984    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.757205 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.757127    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.757205 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.757167    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.757363 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.757339    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.758971 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.758919    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.759404 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.759385    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.820408 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.820379    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdjc\" (UniqueName: \"kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc\") pod \"premium-w1-ef9633-kserve-757b57bfbc-f62wr\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") " pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:20.826258 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:20.826235    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:21.209143 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:58:21.209104    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod552ff4e2_6eb5_45cd_8268_a3595f91dce1.slice/crio-338755b788f4f90b5ed1bc620fef15c3a1e63bf4ef968430ea0c588eb9d95417 WatchSource:0}: Error finding container 338755b788f4f90b5ed1bc620fef15c3a1e63bf4ef968430ea0c588eb9d95417: Status 404 returned error can't find the container with id 338755b788f4f90b5ed1bc620fef15c3a1e63bf4ef968430ea0c588eb9d95417
Nov 03 17:58:21.211293 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:21.211264    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"]
Nov 03 17:58:22.182154 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:22.182119    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" event={"ID":"552ff4e2-6eb5-45cd-8268-a3595f91dce1","Type":"ContainerStarted","Data":"e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412"}
Nov 03 17:58:22.182154 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:22.182163    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" event={"ID":"552ff4e2-6eb5-45cd-8268-a3595f91dce1","Type":"ContainerStarted","Data":"338755b788f4f90b5ed1bc620fef15c3a1e63bf4ef968430ea0c588eb9d95417"}
Nov 03 17:58:22.301107 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:22.301052    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podStartSLOduration=2.301033582 podStartE2EDuration="2.301033582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:22.300070385 +0000 UTC m=+757.530395456" watchObservedRunningTime="2026-11-03 17:58:22.301033582 +0000 UTC m=+757.531358650"
Nov 03 17:58:22.827250 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:22.827215    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:24.520876 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:58:24.520843    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:58:26.736656 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:26.736623    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:58:30.826631 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:30.826596    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:30.845104 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:30.845078    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:31.257040 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:31.256967    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 17:58:50.545593 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:58:50.545555    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:58:54.521653 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:58:54.521615    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:59:19.953649 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:19.953582    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"]
Nov 03 17:59:19.957828 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:19.957811    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.017569 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.017538    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.018278 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.017604    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.018278 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.017634    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.018278 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.017878    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkq45\" (UniqueName: \"kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.018278 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.017953    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.018278 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.018004    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.062592 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.062570    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"premium-w5-2b4f46-kserve-self-signed-certs\""
Nov 03 17:59:20.062592 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.062583    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"openshift-service-ca.crt\""
Nov 03 17:59:20.063230 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.063206    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"kube-root-ca.crt\""
Nov 03 17:59:20.063336 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.063228    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"default-dockercfg-hgn5n\""
Nov 03 17:59:20.065384 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.065365    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"]
Nov 03 17:59:20.122078 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122043    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122283 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122118    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122283 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122165    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122283 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122224    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkq45\" (UniqueName: \"kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122283 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122275    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122324    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122761 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122700    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122953 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122794    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.122953 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.122870    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.124771 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.124750    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.125029 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.125012    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.184822 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.184791    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkq45\" (UniqueName: \"kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45\") pod \"premium-w5-2b4f46-kserve-6d477c649-k7n56\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") " pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.303553 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.303464    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:20.684493 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:59:20.684468    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1541ddfd_21ad_420c_af7e_45604fa1f28d.slice/crio-1c0060d31a7624e73a79e48d5e2f0765d0658203d4d0969f2e64bc7e26f69fd6 WatchSource:0}: Error finding container 1c0060d31a7624e73a79e48d5e2f0765d0658203d4d0969f2e64bc7e26f69fd6: Status 404 returned error can't find the container with id 1c0060d31a7624e73a79e48d5e2f0765d0658203d4d0969f2e64bc7e26f69fd6
Nov 03 17:59:20.687202 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:20.687183    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"]
Nov 03 17:59:21.539506 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:21.539474    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" event={"ID":"1541ddfd-21ad-420c-af7e-45604fa1f28d","Type":"ContainerStarted","Data":"125c2973d27f73b8e3bf8817e82b89d404c2befe2c908bb316b8e8f0319c208f"}
Nov 03 17:59:21.539506 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:21.539519    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" event={"ID":"1541ddfd-21ad-420c-af7e-45604fa1f28d","Type":"ContainerStarted","Data":"1c0060d31a7624e73a79e48d5e2f0765d0658203d4d0969f2e64bc7e26f69fd6"}
Nov 03 17:59:21.659157 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:21.659108    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podStartSLOduration=2.659094153 podStartE2EDuration="2.659094153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:21.658011531 +0000 UTC m=+816.888336602" watchObservedRunningTime="2026-11-03 17:59:21.659094153 +0000 UTC m=+816.889419226"
Nov 03 17:59:22.304185 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:22.304154    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:24.520281 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:59:24.520204    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:59:26.682740 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:26.682684    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:59:30.303986 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:30.303950    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:30.316552 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:30.316527    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:30.610785 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:30.610756    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 17:59:54.521202 ip-10-0-141-108 kubenswrapper[6167]: E1103 17:59:54.521168    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 17:59:56.603698 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:56.603661    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 17:59:58.822421 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.822384    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"]
Nov 03 17:59:58.826509 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.826490    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:58.883276 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.883232    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"openshift-service-ca.crt\""
Nov 03 17:59:58.930376 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.930346    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"premium-w3-0adcbc-kserve-self-signed-certs\""
Nov 03 17:59:58.930558 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.930419    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"kube-root-ca.crt\""
Nov 03 17:59:58.930558 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.930463    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"default-dockercfg-tcxw9\""
Nov 03 17:59:58.933013 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.932989    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"]
Nov 03 17:59:58.978128 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.978096    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:58.978303 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.978148    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:58.978303 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.978167    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:58.978381 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.978354    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:58.978414 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.978390    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh9w\" (UniqueName: \"kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:58.978467 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:58.978451    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079431 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079352    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079431 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079391    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh9w\" (UniqueName: \"kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079645 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079489    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079645 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079536    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079855 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079828    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079967 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079892    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079967 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079898    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079967 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079925    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.079967 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.079957    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.082108 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.082084    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.082402 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.082384    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.141034 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.141002    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh9w\" (UniqueName: \"kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w\") pod \"premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") " pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.141202 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.141189    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 17:59:59.534054 ip-10-0-141-108 kubenswrapper[6167]: W1103 17:59:59.534019    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08886127_238f_4161_b547_897e45db3c05.slice/crio-4fe00b8d04bb733b4ac8efb7253b6358fb01b80ecb870d819f947c6d3476841a WatchSource:0}: Error finding container 4fe00b8d04bb733b4ac8efb7253b6358fb01b80ecb870d819f947c6d3476841a: Status 404 returned error can't find the container with id 4fe00b8d04bb733b4ac8efb7253b6358fb01b80ecb870d819f947c6d3476841a
Nov 03 17:59:59.536241 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.536212    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"]
Nov 03 17:59:59.784444 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.784350    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" event={"ID":"08886127-238f-4161-b547-897e45db3c05","Type":"ContainerStarted","Data":"9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd"}
Nov 03 17:59:59.784444 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.784398    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" event={"ID":"08886127-238f-4161-b547-897e45db3c05","Type":"ContainerStarted","Data":"4fe00b8d04bb733b4ac8efb7253b6358fb01b80ecb870d819f947c6d3476841a"}
Nov 03 17:59:59.902630 ip-10-0-141-108 kubenswrapper[6167]: I1103 17:59:59.902583    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podStartSLOduration=1.9025709979999998 podStartE2EDuration="1.902570998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:59.902241259 +0000 UTC m=+855.132566325" watchObservedRunningTime="2026-11-03 17:59:59.902570998 +0000 UTC m=+855.132896069"
Nov 03 18:00:00.141892 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:00.141849    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:00:09.142394 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:09.142351    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:00:09.160910 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:09.160884    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:00:09.858604 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:09.858574    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:00:21.419123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:21.419084    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"]
Nov 03 18:00:21.419620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:21.419395    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" containerID="cri-o://da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d" gracePeriod=30
Nov 03 18:00:24.502595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:24.502553    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r597d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:00:24.503020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:24.502642    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:00:24.523306 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:00:24.523266    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:00:26.684850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.684825    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 18:00:26.797893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.797802    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.797893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.797841    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798129 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.797922    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798129 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798034    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798129 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798090    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798142    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:00:26.798283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798211    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798228    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:00:26.798283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798244    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798299    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:00:26.798579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798335    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798392    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data\") pod \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\" (UID: \"87b57a48-4012-4b73-9a5c-e69fde62eaf7\") "
Nov 03 18:00:26.798579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798437    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:26.798823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.798675    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:00:26.799081 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.799058    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.799179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.799114    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.799179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.799131    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.799179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.799147    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.799179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.799162    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.800395 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.800369    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:00:26.800496 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.800423    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "kube-api-access-mplxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:26.800496 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.800455    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:26.800496 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.800463    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy" pod "87b57a48-4012-4b73-9a5c-e69fde62eaf7" (UID: "87b57a48-4012-4b73-9a5c-e69fde62eaf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:00:26.900583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.900548    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplxt\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-kube-api-access-mplxt\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.900583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.900578    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.900583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.900590    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.900818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.900607    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b57a48-4012-4b73-9a5c-e69fde62eaf7-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:00:26.954605 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.954571    6167 generic.go:356] "Generic (PLEG): container finished" podID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerID="da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d" exitCode=0
Nov 03 18:00:26.954754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.954626    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"
Nov 03 18:00:26.954754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.954645    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" event={"ID":"87b57a48-4012-4b73-9a5c-e69fde62eaf7","Type":"ContainerDied","Data":"da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d"}
Nov 03 18:00:26.954754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.954700    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d" event={"ID":"87b57a48-4012-4b73-9a5c-e69fde62eaf7","Type":"ContainerDied","Data":"4333a7e167ef0d16cb7c41b67acc249fce6c882b9cd52312241c58a20c660b47"}
Nov 03 18:00:26.954754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.954733    6167 scope.go:122] "RemoveContainer" containerID="da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d"
Nov 03 18:00:26.965627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.965602    6167 scope.go:122] "RemoveContainer" containerID="037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"
Nov 03 18:00:26.977278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.977259    6167 scope.go:122] "RemoveContainer" containerID="da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d"
Nov 03 18:00:26.977571 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:00:26.977554    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d\": container with ID starting with da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d not found: ID does not exist" containerID="da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d"
Nov 03 18:00:26.977629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.977581    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d"} err="failed to get container status \"da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d\": rpc error: code = NotFound desc = could not find container \"da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d\": container with ID starting with da21aaf07bf99042485b5d64ab81d1c5cc98e9a25d0896ae4848f0f3e55fa21d not found: ID does not exist"
Nov 03 18:00:26.977629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.977598    6167 scope.go:122] "RemoveContainer" containerID="037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"
Nov 03 18:00:26.977860 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:00:26.977845    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22\": container with ID starting with 037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22 not found: ID does not exist" containerID="037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"
Nov 03 18:00:26.977924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:26.977863    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22"} err="failed to get container status \"037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22\": rpc error: code = NotFound desc = could not find container \"037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22\": container with ID starting with 037fec895ef8c6bbce11a0eb74dc0d454973ac0e5054ab5ef53a6db3b8520e22 not found: ID does not exist"
Nov 03 18:00:27.124227 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:27.124195    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"]
Nov 03 18:00:27.128658 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:27.128632    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r597d"]
Nov 03 18:00:28.930740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:28.930672    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" path="/var/lib/kubelet/pods/87b57a48-4012-4b73-9a5c-e69fde62eaf7/volumes"
Nov 03 18:00:31.968959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:31.968922    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:00:45.033104 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:45.033063    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:00:45.035589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:45.035563    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:00:54.482961 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.482888    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"]
Nov 03 18:00:54.483408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.483394    6167 cpu_manager.go:414] "containerMap: removing container" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy"
Nov 03 18:00:54.483451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.483409    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy"
Nov 03 18:00:54.483451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.483435    6167 cpu_manager.go:414] "containerMap: removing container" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy"
Nov 03 18:00:54.483451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.483442    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy"
Nov 03 18:00:54.483545 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.483537    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy"
Nov 03 18:00:54.483579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.483545    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b57a48-4012-4b73-9a5c-e69fde62eaf7" containerName="istio-proxy"
Nov 03 18:00:54.488270 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.488245    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.521088 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:00:54.521057    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:00:54.542228 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.542204    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ef9633-openshift-default-dockercfg-49fqv\""
Nov 03 18:00:54.595261 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.595228    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"]
Nov 03 18:00:54.600633 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.600611    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.600770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.600639    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.600836 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.600777    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.600836 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.600804    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.600959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.600923    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.601015 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.600975    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrs6\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.601069 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.601037    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.601160 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.601133    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.601218 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.601187    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702165 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702132    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702165 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702169    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702228    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702256    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrs6\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702275    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702302    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702339    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702416    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702439    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702621    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.702844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.702752    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.703029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.703004    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.703097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.703053    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.703295 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.703269    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.704964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.704944    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.705058 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.705039    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.766471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.766389    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.766640 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.766609    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrs6\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:54.801305 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:54.801275    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:55.276308 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:00:55.276277    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87b6388_baa5_4920_910a_87b7a8589191.slice/crio-9bd4bbe9f0efdc6a89f5d3c8be9f4d6812771b18268774dbbd88ad9a4029920b WatchSource:0}: Error finding container 9bd4bbe9f0efdc6a89f5d3c8be9f4d6812771b18268774dbbd88ad9a4029920b: Status 404 returned error can't find the container with id 9bd4bbe9f0efdc6a89f5d3c8be9f4d6812771b18268774dbbd88ad9a4029920b
Nov 03 18:00:55.277647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:55.277618    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"]
Nov 03 18:00:55.279352 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:55.279312    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:00:55.279440 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:55.279396    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:00:55.279482 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:55.279441    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:00:56.165683 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:56.165641    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" event={"ID":"e87b6388-baa5-4920-910a-87b7a8589191","Type":"ContainerStarted","Data":"a13b7cf80164a11afd3e2985f94f98cff3c1825d5bd2b3915631290f06bcf2d1"}
Nov 03 18:00:56.166400 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:56.166350    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" event={"ID":"e87b6388-baa5-4920-910a-87b7a8589191","Type":"ContainerStarted","Data":"9bd4bbe9f0efdc6a89f5d3c8be9f4d6812771b18268774dbbd88ad9a4029920b"}
Nov 03 18:00:56.353319 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:56.353258    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podStartSLOduration=2.35323875 podStartE2EDuration="2.35323875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:56.350678352 +0000 UTC m=+911.581003493" watchObservedRunningTime="2026-11-03 18:00:56.35323875 +0000 UTC m=+911.583563824"
Nov 03 18:00:56.802316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:56.802277    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:00:56.804243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:56.804222    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 18:00:56.804306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:56.804268    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 18:00:57.802245 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:57.802208    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 18:00:57.802656 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:57.802278    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 18:00:58.802108 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:58.802074    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 18:00:58.802289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:58.802137    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 18:00:59.801801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:59.801764    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 18:00:59.802020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:00:59.801843    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 18:01:00.802293 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:00.802260    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 18:01:00.802702 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:00.802322    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 18:01:02.745900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:02.745847    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:01:02.746610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:02.746587    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:01:02.747668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:02.747642    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:01:12.612398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:12.612366    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:01:21.919461 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:21.919421    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"]
Nov 03 18:01:21.923995 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:21.923970    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041532 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041498    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041553    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041635    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041671    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041747    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041797    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhw8\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041819    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.041853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041839    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.042005 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.041865    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.078804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.078775    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"]
Nov 03 18:01:22.142701 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142668    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.142891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142742    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.142891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142777    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhw8\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.142891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142797    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.142891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142816    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.142891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142844    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142920    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142962    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.142991    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.143209    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.143252    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.143299    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.143562    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.143735 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.143689    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.145305 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.145278    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.145687 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.145671    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.205419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.205343    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.205619 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.205600    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhw8\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8\") pod \"maas-default-gateway-openshift-default-6dcf757899-6xhmj\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.238390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.238362    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:22.680363 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:01:22.680334    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265c6d6_17a6_4705_9b2f_40407849d35c.slice/crio-fec00bf2d070b0f5a464f6418a00bcae3caabd3a6f1bbc6a72f46607ba0be843 WatchSource:0}: Error finding container fec00bf2d070b0f5a464f6418a00bcae3caabd3a6f1bbc6a72f46607ba0be843: Status 404 returned error can't find the container with id fec00bf2d070b0f5a464f6418a00bcae3caabd3a6f1bbc6a72f46607ba0be843
Nov 03 18:01:22.682944 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.682906    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:22.683038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.682991    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:22.683038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.683032    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:22.683460 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:22.683435    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"]
Nov 03 18:01:23.364470 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:23.364430    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" event={"ID":"3265c6d6-17a6-4705-9b2f-40407849d35c","Type":"ContainerStarted","Data":"e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d"}
Nov 03 18:01:23.365010 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:23.364502    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" event={"ID":"3265c6d6-17a6-4705-9b2f-40407849d35c","Type":"ContainerStarted","Data":"fec00bf2d070b0f5a464f6418a00bcae3caabd3a6f1bbc6a72f46607ba0be843"}
Nov 03 18:01:23.491464 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:23.491403    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podStartSLOduration=2.491386142 podStartE2EDuration="2.491386142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:23.489686181 +0000 UTC m=+938.720011264" watchObservedRunningTime="2026-11-03 18:01:23.491386142 +0000 UTC m=+938.721711215"
Nov 03 18:01:24.243744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:24.239108    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:24.251744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:24.244775    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:24.251744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:24.244832    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:24.520656 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:01:24.520570    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:01:25.241107 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:25.241068    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:25.242120 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:25.242089    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:26.246743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.245039    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:26.246743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.245097    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:26.566297 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.566263    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"]
Nov 03 18:01:26.571289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.571271    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.624830 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.624803    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a86228-openshift-default-dockercfg-pqbnk\""
Nov 03 18:01:26.678555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.678514    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"]
Nov 03 18:01:26.702086 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702050    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702119    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702329 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702257    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702329 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702318    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702443    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702483    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702587 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702564    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btsg\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702645 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702624    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.702786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.702767    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804084    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804128    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804150    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btsg\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804176    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804210    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804244    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804288    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804348    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804392    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804616    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804741    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804755    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.804802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.804787    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.805292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.805275    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.806791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.806763    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.806902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.806812    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.867313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.867202    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.867452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.867331    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btsg\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:26.888601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:26.888571    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:27.258134 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.258044    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:27.265765 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.258651    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:27.367802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.367774    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"]
Nov 03 18:01:27.370331 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:01:27.370308    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda424743b_e4c8_4e6e_ad03_5edf1d6fe60a.slice/crio-efb00a55382529a0fc7b3ca2b541fb74b4cfaa1e1c935975853ca1fe4c2ca177 WatchSource:0}: Error finding container efb00a55382529a0fc7b3ca2b541fb74b4cfaa1e1c935975853ca1fe4c2ca177: Status 404 returned error can't find the container with id efb00a55382529a0fc7b3ca2b541fb74b4cfaa1e1c935975853ca1fe4c2ca177
Nov 03 18:01:27.373036 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.373002    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:27.373140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.373077    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:27.373140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.373122    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:27.392104 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:27.392079    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" event={"ID":"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a","Type":"ContainerStarted","Data":"efb00a55382529a0fc7b3ca2b541fb74b4cfaa1e1c935975853ca1fe4c2ca177"}
Nov 03 18:01:28.249741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.242376    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:28.249741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.242455    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:28.402346 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.402311    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" event={"ID":"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a","Type":"ContainerStarted","Data":"17a0efcbfb837a885d8c001ad03634508a3d74b7202852a87f9d01ebf505f215"}
Nov 03 18:01:28.526942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.526834    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podStartSLOduration=2.526819571 podStartE2EDuration="2.526819571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:28.524620509 +0000 UTC m=+943.754945581" watchObservedRunningTime="2026-11-03 18:01:28.526819571 +0000 UTC m=+943.757144640"
Nov 03 18:01:28.888792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.888755    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:28.890529 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.890499    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:28.890653 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:28.890556    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 18:01:29.245920 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:29.245822    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:29.246177 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:29.246146    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:29.889876 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:29.889834    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:29.890289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:29.889909    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 18:01:30.240946 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:30.240848    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 18:01:30.241200 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:30.241171    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 18:01:30.893233 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:30.893197    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:30.893865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:30.893824    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 18:01:31.903742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:31.902858    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:31.903742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:31.902945    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 18:01:32.259822 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.251803    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6xhmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:01:32.259822 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.251882    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:01:32.610104 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.610069    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:32.611435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.611307    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:32.614925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.614903    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:01:32.890012 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.889923    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:32.890012 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:32.889988    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 18:01:33.889593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:33.889554    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 18:01:33.890084 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:33.889617    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 18:01:35.364768 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:35.364712    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:35.365370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:35.365347    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:35.366207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:35.366188    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:01:43.309945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:43.309911    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:01:54.521878 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:01:54.521836    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:01:54.980583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:54.980545    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"]
Nov 03 18:01:54.984953 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:54.984937    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.063429 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.063389    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"]
Nov 03 18:01:55.123312 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123281    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsvx\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123502 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123425    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123531    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123575    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123669 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123639    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123676    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123795 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123782    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123838 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123810    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.123884 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.123835    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225072 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225040    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsvx\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225190    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225229    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225252    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225445 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225366    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225445 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225406    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225587 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225475    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225587 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225499    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225587 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225529    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225593    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225667    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225695    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.225938 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.225920    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.226179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.226160    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.227896 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.227873    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.228005 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.227990    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.292476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.292387    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.292476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.292474    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsvx\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.299289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.299256    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:55.735055 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.735010    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"]
Nov 03 18:01:55.757205 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:01:55.757176    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8c00df_c0fe_41ea_b486_88c3d0e1be36.slice/crio-164900d110b3f4daae0701dfffa10e7cbaebecf91c6dfe6a31d0b12632d7f625 WatchSource:0}: Error finding container 164900d110b3f4daae0701dfffa10e7cbaebecf91c6dfe6a31d0b12632d7f625: Status 404 returned error can't find the container with id 164900d110b3f4daae0701dfffa10e7cbaebecf91c6dfe6a31d0b12632d7f625
Nov 03 18:01:55.759679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.759646    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:55.759795 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.759766    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:55.759836 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:55.759802    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:01:56.663211 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:56.663176    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" event={"ID":"cb8c00df-c0fe-41ea-b486-88c3d0e1be36","Type":"ContainerStarted","Data":"34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e"}
Nov 03 18:01:56.663390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:56.663232    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" event={"ID":"cb8c00df-c0fe-41ea-b486-88c3d0e1be36","Type":"ContainerStarted","Data":"164900d110b3f4daae0701dfffa10e7cbaebecf91c6dfe6a31d0b12632d7f625"}
Nov 03 18:01:56.862228 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:56.862175    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podStartSLOduration=2.862160137 podStartE2EDuration="2.862160137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:56.860358919 +0000 UTC m=+972.090683974" watchObservedRunningTime="2026-11-03 18:01:56.862160137 +0000 UTC m=+972.092485280"
Nov 03 18:01:57.299844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:57.299808    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:01:57.301637 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:57.301610    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 18:01:57.301776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:57.301676    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 18:01:58.299756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:58.299694    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 18:01:58.300145 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:58.299775    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 18:01:59.299829 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:59.299788    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 18:01:59.300222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:01:59.299850    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 18:02:00.299893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:00.299861    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 18:02:00.300279 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:00.299913    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 18:02:01.300580 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:01.300509    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 18:02:01.301166 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:01.300583    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 18:02:02.410304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:02.410061    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:02:02.411275 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:02.411011    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:02:02.412019 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:02.411996    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:02:08.557288 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:08.557254    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:02:14.306612 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.306577    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"]
Nov 03 18:02:14.310682 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.310665    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.366796 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.366767    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"]
Nov 03 18:02:14.374950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.374921    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.375121 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.375002    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.375121 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.375106    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.375240 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.375162    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjzz\" (UniqueName: \"kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.375296 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.375261    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.375351 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.375317    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.414601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.414577    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"openshift-service-ca.crt\""
Nov 03 18:02:14.414762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.414612    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"distinct-w6-aab753-kserve-self-signed-certs\""
Nov 03 18:02:14.414762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.414645    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"kube-root-ca.crt\""
Nov 03 18:02:14.414762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.414675    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"default-dockercfg-b6977\""
Nov 03 18:02:14.477134 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.477099    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.477538 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.477507    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.477680 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.477583    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.477680 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.477673    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.477828 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.477737    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjzz\" (UniqueName: \"kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.477904 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.477832    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.478566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.478163    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.478566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.478269    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.478566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.478367    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.480855 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.480834    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.481969 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.481950    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.541057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.541025    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjzz\" (UniqueName: \"kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz\") pod \"distinct-w6-aab753-kserve-6fc576b9b6-c29hg\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:14.624665 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:14.624638    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:15.021850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:15.021826    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"]
Nov 03 18:02:15.804888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:15.804851    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" event={"ID":"cfb1bcf2-cf99-4255-933f-48a10b71f9ba","Type":"ContainerStarted","Data":"1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f"}
Nov 03 18:02:15.805272 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:15.804898    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" event={"ID":"cfb1bcf2-cf99-4255-933f-48a10b71f9ba","Type":"ContainerStarted","Data":"6f2783db5b0c7be9f79532a64d98ba0793135c48baabfa2d042809069e26045c"}
Nov 03 18:02:15.927307 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:15.927249    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podStartSLOduration=1.927234699 podStartE2EDuration="1.927234699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:15.924962362 +0000 UTC m=+991.155287435" watchObservedRunningTime="2026-11-03 18:02:15.927234699 +0000 UTC m=+991.157559772"
Nov 03 18:02:16.624880 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:16.624851    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:24.521516 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:02:24.521428    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:02:24.625035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:24.625004    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:24.637769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:24.637739    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:24.879131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:24.879103    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:02:32.617788 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.617686    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"]
Nov 03 18:02:32.622833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.622809    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.680583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.680553    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"]
Nov 03 18:02:32.694508 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.694480    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.694650 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.694521    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4624\" (UniqueName: \"kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.694650 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.694626    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.694786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.694654    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.694786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.694669    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.694910 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.694884    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.727107 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.727075    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"distinct-w1-ef9633-kserve-self-signed-certs\""
Nov 03 18:02:32.796330 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796292    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796336    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4624\" (UniqueName: \"kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796392    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796424    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796446    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796521    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796906 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796799    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.796906 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796809    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.797006 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.796902    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.798794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.798768    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.799042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.799026    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.866478 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.866448    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4624\" (UniqueName: \"kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624\") pod \"distinct-w1-ef9633-kserve-9db868c58-t8vcc\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:32.938448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:32.938369    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:33.355137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:33.355113    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"]
Nov 03 18:02:33.921956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:33.921924    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" event={"ID":"df67e709-136e-428c-af59-abd8a1671d30","Type":"ContainerStarted","Data":"9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab"}
Nov 03 18:02:33.922337 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:33.921967    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" event={"ID":"df67e709-136e-428c-af59-abd8a1671d30","Type":"ContainerStarted","Data":"9e90595ca591078b79d9cc9b37b54f189969035a1f6f871ed8aebe961f19e508"}
Nov 03 18:02:33.939079 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:33.939055    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:34.045140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:34.045087    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podStartSLOduration=2.045074076 podStartE2EDuration="2.045074076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:34.04318925 +0000 UTC m=+1009.273514333" watchObservedRunningTime="2026-11-03 18:02:34.045074076 +0000 UTC m=+1009.275399147"
Nov 03 18:02:38.647939 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:38.647907    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:02:42.939509 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:42.939475    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:42.953593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:42.953564    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:42.995211 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:02:42.995189    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:02:54.521022 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:02:54.520986    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:03:07.268476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:07.268438    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:03:10.256450 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.256415    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"]
Nov 03 18:03:10.260703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.260686    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.324141 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.324106    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"]
Nov 03 18:03:10.328052 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.328034    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.332434 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.332407    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f79c02-openshift-default-dockercfg-qxdds\""
Nov 03 18:03:10.361051 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361021    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361066    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txbn\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361279 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361252    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361327 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361301    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361392    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361515 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361500    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361538    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361593    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.361677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.361645    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.376601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.376561    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"]
Nov 03 18:03:10.393462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.393427    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"]
Nov 03 18:03:10.463219 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463182    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.463404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463296    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.463404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463325    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.463404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463356    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.463404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463377    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.463404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463393    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.463678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463408    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.463678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463445    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.463678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463627    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.463868 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463775    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.463927 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463886    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.463967 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463941    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.464153 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.463809    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr978\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.464471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.464443    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.467374 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.467345    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.477102 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477071    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txbn\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.477250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477157    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.477250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477240    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.477364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477280    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.477364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477310    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.477364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477357    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.477521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477424    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.477924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.477901    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.478287 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.478262    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.480448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.480421    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.525421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.525353    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.541866 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.541814    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txbn\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.574629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.574596    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:10.578170 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578137    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578334 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578177    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578334 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578267    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578334 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578329    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578503 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578365    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578503 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578389    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578503 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578415    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578503    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr978\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578562    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578574    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578634    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.578954 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.578930    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.579498 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.579080    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.579498 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.579244    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.581167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.581144    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.581530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.581509    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.642376 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.642312    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr978\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.652518 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.652483    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:10.944916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:10.944881    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:11.002944 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:03:11.002913    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3a90e0_1e22_40c7_8b52_d732ee0c6903.slice/crio-bf1347f31220e0d9faa82cb914d071220606e6dbfc80ba132c68e98d4d790309 WatchSource:0}: Error finding container bf1347f31220e0d9faa82cb914d071220606e6dbfc80ba132c68e98d4d790309: Status 404 returned error can't find the container with id bf1347f31220e0d9faa82cb914d071220606e6dbfc80ba132c68e98d4d790309
Nov 03 18:03:11.004181 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.004151    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"]
Nov 03 18:03:11.006430 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.006395    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:03:11.006536 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.006472    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:03:11.006536 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.006520    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:03:11.169745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.168796    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" event={"ID":"9c3a90e0-1e22-40c7-8b52-d732ee0c6903","Type":"ContainerStarted","Data":"e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d"}
Nov 03 18:03:11.169745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.168862    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" event={"ID":"9c3a90e0-1e22-40c7-8b52-d732ee0c6903","Type":"ContainerStarted","Data":"bf1347f31220e0d9faa82cb914d071220606e6dbfc80ba132c68e98d4d790309"}
Nov 03 18:03:11.291769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.291665    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podStartSLOduration=1.291651735 podStartE2EDuration="1.291651735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:11.290219509 +0000 UTC m=+1046.520544582" watchObservedRunningTime="2026-11-03 18:03:11.291651735 +0000 UTC m=+1046.521976860"
Nov 03 18:03:11.330426 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:03:11.330393    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b89db3_b00c_49cd_af19_08b5d49b6712.slice/crio-ae2fc21196375870080e3bcfcb8d252abf8fd07ed0922d7a1836fd7a925c750d WatchSource:0}: Error finding container ae2fc21196375870080e3bcfcb8d252abf8fd07ed0922d7a1836fd7a925c750d: Status 404 returned error can't find the container with id ae2fc21196375870080e3bcfcb8d252abf8fd07ed0922d7a1836fd7a925c750d
Nov 03 18:03:11.333290 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.333252    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:03:11.333380 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.333341    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:03:11.333433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.333381    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:03:11.345127 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.345101    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"]
Nov 03 18:03:11.576294 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:11.576167    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:12.179070 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.179000    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" event={"ID":"66b89db3-b00c-49cd-af19-08b5d49b6712","Type":"ContainerStarted","Data":"9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0"}
Nov 03 18:03:12.179281 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.179092    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" event={"ID":"66b89db3-b00c-49cd-af19-08b5d49b6712","Type":"ContainerStarted","Data":"ae2fc21196375870080e3bcfcb8d252abf8fd07ed0922d7a1836fd7a925c750d"}
Nov 03 18:03:12.303111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.303047    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podStartSLOduration=2.303033542 podStartE2EDuration="2.303033542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:12.301104176 +0000 UTC m=+1047.531429248" watchObservedRunningTime="2026-11-03 18:03:12.303033542 +0000 UTC m=+1047.533358613"
Nov 03 18:03:12.599741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.597018    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 18:03:12.599741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.597075    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 18:03:12.954799 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.953778    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:12.959738 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.957858    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 18:03:12.959738 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:12.957959    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 18:03:13.581326 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:13.581291    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 18:03:13.581949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:13.581913    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 18:03:13.950802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:13.949156    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 18:03:13.950802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:13.949230    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 18:03:14.575094 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:14.575055    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.575387 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:14.575359    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 18:03:14.950801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:14.948108    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.950801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:14.948166    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 18:03:15.583747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:15.575180    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.583747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:15.575248    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 18:03:15.950801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:15.950148    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.950801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:15.950212    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 18:03:16.579848 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:16.579808    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.580155 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:16.580124    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 18:03:16.952230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:16.952134    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.952840 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:16.952810    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 18:03:17.576323 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:17.576287    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.576524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:17.576344    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 18:03:17.945736 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:17.945616    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.945736 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:17.945668    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 18:03:18.599861 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:18.599827    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:18.600711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:18.600688    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:18.601549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:18.601522    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:03:18.953387 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:18.952780    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:18.953582 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:18.953561    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:18.954605 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:18.954586    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:03:24.521552 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:03:24.521516    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:03:46.905066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:03:46.905026    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:03:54.521696 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:03:54.521612    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:04:17.246876 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:17.246833    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:04:24.522004 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:04:24.521965    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:04:27.795161 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.795127    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"]
Nov 03 18:04:27.799450 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.799432    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.815126 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.815101    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.815245 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.815164    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.815310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.815285    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.815398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.815380    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.815463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.815408    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.815522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.815503    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj7h\" (UniqueName: \"kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.905211 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.905178    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aab753\"/\"distinct-2-w6-aab753-kserve-self-signed-certs\""
Nov 03 18:04:27.907683 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.907661    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"]
Nov 03 18:04:27.916795 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.916774    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.916863 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.916825    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.916916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.916900    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.917013 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.916992    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.917066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.917025    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.917066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.917044    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj7h\" (UniqueName: \"kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.917244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.917224    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.917382 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.917360    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.917435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.917369    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.919113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.919096    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.919400 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.919382    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.985092 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.985063    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj7h\" (UniqueName: \"kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h\") pod \"distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") " pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:27.995406 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.995370    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"]
Nov 03 18:04:27.999768 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:27.999713    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018509 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018482    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xf2\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018539    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018701 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018683    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018783 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018729    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018843 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018828    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018881 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018857    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.018986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018972    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.019021 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.018994    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.019062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.019049    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.082776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.082744    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"]
Nov 03 18:04:28.111770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.111710    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:28.120071 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120042    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120079    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120113    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120158    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xf2\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120192    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120285    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120308    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120367    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120392    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120472    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120517    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120840 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120811    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.120896 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.120872    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.121293 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.121267    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.122625 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.122605    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.122914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.122896    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.190138 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.190100    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.190852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.190785    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xf2\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.313660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.313629    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:28.546582 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:04:28.546545    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73a3c197_2658_465b_9271_d7367e9e79e9.slice/crio-781117bb64ea1b6c2b879e41251ed568b3a859133be5ef41cc9957768b10f9ab WatchSource:0}: Error finding container 781117bb64ea1b6c2b879e41251ed568b3a859133be5ef41cc9957768b10f9ab: Status 404 returned error can't find the container with id 781117bb64ea1b6c2b879e41251ed568b3a859133be5ef41cc9957768b10f9ab
Nov 03 18:04:28.550620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.550592    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"]
Nov 03 18:04:28.698819 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:04:28.698776    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89fdd5f_25ed_4b9f_9a77_56c170f9c17a.slice/crio-2efbc76655307b74930b7850c7e3f798e54231fd0145b03a964be43074c6760d WatchSource:0}: Error finding container 2efbc76655307b74930b7850c7e3f798e54231fd0145b03a964be43074c6760d: Status 404 returned error can't find the container with id 2efbc76655307b74930b7850c7e3f798e54231fd0145b03a964be43074c6760d
Nov 03 18:04:28.700958 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.700926    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" event={"ID":"73a3c197-2658-465b-9271-d7367e9e79e9","Type":"ContainerStarted","Data":"ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce"}
Nov 03 18:04:28.701072 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.700978    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" event={"ID":"73a3c197-2658-465b-9271-d7367e9e79e9","Type":"ContainerStarted","Data":"781117bb64ea1b6c2b879e41251ed568b3a859133be5ef41cc9957768b10f9ab"}
Nov 03 18:04:28.701553 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.701528    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:04:28.701599 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.701588    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:04:28.701637 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.701618    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:04:28.701804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.701784    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"]
Nov 03 18:04:28.821469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:28.821410    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podStartSLOduration=1.821390813 podStartE2EDuration="1.821390813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:28.818478227 +0000 UTC m=+1124.048803299" watchObservedRunningTime="2026-11-03 18:04:28.821390813 +0000 UTC m=+1124.051715863"
Nov 03 18:04:29.112633 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:29.112603    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:29.709405 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:29.709362    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" event={"ID":"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a","Type":"ContainerStarted","Data":"a21346ba92ab4667edbe008dacbf8934aaaa5fe23de8565c14b2ff12ebd9e03f"}
Nov 03 18:04:29.709613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:29.709427    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" event={"ID":"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a","Type":"ContainerStarted","Data":"2efbc76655307b74930b7850c7e3f798e54231fd0145b03a964be43074c6760d"}
Nov 03 18:04:29.883481 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:29.883414    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podStartSLOduration=2.883397037 podStartE2EDuration="2.883397037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:29.882140557 +0000 UTC m=+1125.112465631" watchObservedRunningTime="2026-11-03 18:04:29.883397037 +0000 UTC m=+1125.113722111"
Nov 03 18:04:30.314301 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:30.314271    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:30.316175 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:30.316149    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:30.316304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:30.316197    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 18:04:31.314080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:31.314039    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:31.314479 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:31.314104    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 18:04:32.314910 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:32.314873    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:32.315343 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:32.314927    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 18:04:33.314647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:33.314608    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:33.314852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:33.314668    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 18:04:34.314367 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:34.314331    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:34.314771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:34.314383    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 18:04:35.314648 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:35.314613    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 18:04:35.315092 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:35.314667    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 18:04:36.318832 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:36.318800    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:36.319329 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:36.319186    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:36.319749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:36.319712    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:04:38.112073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:38.112038    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:38.125004 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:38.124981    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:38.802992 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:38.802965    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:04:49.629756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:49.629694    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:04:51.202458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.202425    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"]
Nov 03 18:04:51.206868 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.206847    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.269758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.269129    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.269758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.269180    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.269758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.269217    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.269758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.269348    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.269758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.269438    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.269758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.269500    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5wcb\" (UniqueName: \"kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.312684 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.312652    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"]
Nov 03 18:04:51.319097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.319064    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"kube-root-ca.crt\""
Nov 03 18:04:51.319310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.319288    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"distinct-2-w2-f79c02-kserve-self-signed-certs\""
Nov 03 18:04:51.319402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.319319    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"openshift-service-ca.crt\""
Nov 03 18:04:51.319402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.319330    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"default-dockercfg-pxq8r\""
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.384965    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.385080    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.385129    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5wcb\" (UniqueName: \"kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.385260    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.385285    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.385318    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.403745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.385818    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.427510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.409743    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.427510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.422541    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.459786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.454387    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.504742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.500280    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.504742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.500638    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5wcb\" (UniqueName: \"kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb\") pod \"distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") " pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:51.520303 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:51.520029    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:52.034330 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:04:52.034286    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0214d7cf_6f58_4889_99d8_f98e8553a400.slice/crio-6b0a341f79021742d0dbe23ec7cb0e7d1fa081d4c6adf92d2bd1fbe8f76c114b WatchSource:0}: Error finding container 6b0a341f79021742d0dbe23ec7cb0e7d1fa081d4c6adf92d2bd1fbe8f76c114b: Status 404 returned error can't find the container with id 6b0a341f79021742d0dbe23ec7cb0e7d1fa081d4c6adf92d2bd1fbe8f76c114b
Nov 03 18:04:52.036184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.036161    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"]
Nov 03 18:04:52.688117 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.688077    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-b2j58-qxzzg"]
Nov 03 18:04:52.692317 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.692286    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.745749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.745707    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jd7fn\""
Nov 03 18:04:52.747509 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.747482    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-b2j58-qxzzg"]
Nov 03 18:04:52.815787 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.815753    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfkq\" (UniqueName: \"kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.815787 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.815803    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.816054 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.815846    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.816054 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.816019    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.899839 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.899804    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" event={"ID":"0214d7cf-6f58-4889-99d8-f98e8553a400","Type":"ContainerStarted","Data":"04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca"}
Nov 03 18:04:52.899839 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.899848    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" event={"ID":"0214d7cf-6f58-4889-99d8-f98e8553a400","Type":"ContainerStarted","Data":"6b0a341f79021742d0dbe23ec7cb0e7d1fa081d4c6adf92d2bd1fbe8f76c114b"}
Nov 03 18:04:52.917693 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.917658    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.917889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.917866    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfkq\" (UniqueName: \"kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.917940 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.917892    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.917940 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.917926    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.918301 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.918280    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.918475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.918455    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.918550 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.918520    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:52.980071 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:52.979991    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfkq\" (UniqueName: \"kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq\") pod \"periodic-gathering-b2j58-qxzzg\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") " pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:53.005862 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:53.005816    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:04:53.020073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:53.020027    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podStartSLOduration=2.020013248 podStartE2EDuration="2.020013248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:53.017634615 +0000 UTC m=+1148.247959691" watchObservedRunningTime="2026-11-03 18:04:53.020013248 +0000 UTC m=+1148.250338320"
Nov 03 18:04:53.379949 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:04:53.379917    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c37162_79cd_4a18_9346_87ccf834737f.slice/crio-232ef7bcfa0ae9d47a852d4b11aa8d81154383a2dfadac6eda794de0557dd54c WatchSource:0}: Error finding container 232ef7bcfa0ae9d47a852d4b11aa8d81154383a2dfadac6eda794de0557dd54c: Status 404 returned error can't find the container with id 232ef7bcfa0ae9d47a852d4b11aa8d81154383a2dfadac6eda794de0557dd54c
Nov 03 18:04:53.383413 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:53.383377    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-b2j58-qxzzg"]
Nov 03 18:04:53.520697 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:53.520669    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:04:53.907194 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:53.907158    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2j58-qxzzg" event={"ID":"57c37162-79cd-4a18-9346-87ccf834737f","Type":"ContainerStarted","Data":"d2173174b84414e92313c6d560a9434ab4c7e6ac771a9341439f66eecddea817"}
Nov 03 18:04:53.907606 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:53.907216    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2j58-qxzzg" event={"ID":"57c37162-79cd-4a18-9346-87ccf834737f","Type":"ContainerStarted","Data":"232ef7bcfa0ae9d47a852d4b11aa8d81154383a2dfadac6eda794de0557dd54c"}
Nov 03 18:04:54.025639 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:04:54.025587    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-b2j58-qxzzg" podStartSLOduration=2.025574193 podStartE2EDuration="2.025574193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:54.023386846 +0000 UTC m=+1149.253711918" watchObservedRunningTime="2026-11-03 18:04:54.025574193 +0000 UTC m=+1149.255899265"
Nov 03 18:04:54.521384 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:04:54.521350    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:05:01.520789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:01.520748    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:05:01.534150 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:01.534117    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:05:01.980704 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:01.980674    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:05:12.099755 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.099704    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"]
Nov 03 18:05:12.105757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.105710    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.198480 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.198446    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"]
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.256661    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.276888    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.276961    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.277102    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.277173    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.277314    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.277357    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.277415    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.278742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.277450    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ck6\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.378996 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.378875    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.379792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.379768    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.380053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.380011    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.380299 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.380279    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ck6\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.380506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.380487    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.384537 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.380829    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.384686 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.380662    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.384853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.384834    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.385074 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.385055    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.385274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.385255    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.385994 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.385975    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.386479 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.386460    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.387501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.387482    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.396120 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.396063    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.402319 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.402291    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.405369 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.405345    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.459753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.448903    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ck6\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.521762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.518372    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:12.749793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:12.731943    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:13.581746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:13.581597    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"]
Nov 03 18:05:13.658748 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:05:13.658149    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e61eebf_fb19_4f5f_80ae_9bd904347052.slice/crio-ab7e6306e1a8397a3b2b1c42235b22d7fcbcc8d3c98c8ad589148750b331ffcb WatchSource:0}: Error finding container ab7e6306e1a8397a3b2b1c42235b22d7fcbcc8d3c98c8ad589148750b331ffcb: Status 404 returned error can't find the container with id ab7e6306e1a8397a3b2b1c42235b22d7fcbcc8d3c98c8ad589148750b331ffcb
Nov 03 18:05:13.680914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:13.679436    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:13.680914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:13.679528    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:13.680914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:13.679575    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:14.394695 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:14.394654    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" event={"ID":"6e61eebf-fb19-4f5f-80ae-9bd904347052","Type":"ContainerStarted","Data":"33947b4d0cf5fe421be106e8693eb4385c6675b24aa4fa839ce6697d60af9e20"}
Nov 03 18:05:14.394937 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:14.394758    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" event={"ID":"6e61eebf-fb19-4f5f-80ae-9bd904347052","Type":"ContainerStarted","Data":"ab7e6306e1a8397a3b2b1c42235b22d7fcbcc8d3c98c8ad589148750b331ffcb"}
Nov 03 18:05:14.621124 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:14.621072    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podStartSLOduration=2.621055547 podStartE2EDuration="2.621055547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:14.570112012 +0000 UTC m=+1169.800437083" watchObservedRunningTime="2026-11-03 18:05:14.621055547 +0000 UTC m=+1169.851380619"
Nov 03 18:05:14.744808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:14.733920    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:14.744808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:14.736315    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:14.744808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:14.736367    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:15.745744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:15.734160    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:15.745744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:15.734239    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:16.747681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:16.747643    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:16.748324 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:16.748289    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:17.734128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:17.734090    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:17.734440 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:17.734409    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:18.763208 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:18.762680    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:18.763208 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:18.762807    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:19.745471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:19.745436    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:19.745775 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:19.745745    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:20.732758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:20.732706    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 18:05:20.733261 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:20.732790    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 18:05:21.739038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:21.739008    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:21.740257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:21.740110    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:21.740816 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:21.740796    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:05:24.527811 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:05:24.527010    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:05:26.543362 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.543326    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"]
Nov 03 18:05:26.550023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.549992    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.657701 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.657671    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"]
Nov 03 18:05:26.716535 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.716493    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.716902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.716884    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717100 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717085    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717292    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717410 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717397    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717502 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717490    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9m5\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717608    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717764 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717748    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.717887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.717874    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819444 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819417    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819462    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819512    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819541    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819597    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819862 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819756    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819862 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819796    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.819862 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819819    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9m5\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.820040 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.819871    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.820040 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.820010    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.820227 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.820179    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.820356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.820335    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.820426 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.820390    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.820426 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.820406    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.821869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.821844    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.822307 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.822285    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.883307 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.883270    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:26.883307 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:26.883303    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9m5\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:27.165245 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:27.165167    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:27.587939 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:05:27.587904    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0459156f_933e_4bb4_88bc_976d5e41c36f.slice/crio-fe84a07c55d9c3977c0e1b85a73ebff0af80dfb9fa20dc3bb521b5515cb873d6 WatchSource:0}: Error finding container fe84a07c55d9c3977c0e1b85a73ebff0af80dfb9fa20dc3bb521b5515cb873d6: Status 404 returned error can't find the container with id fe84a07c55d9c3977c0e1b85a73ebff0af80dfb9fa20dc3bb521b5515cb873d6
Nov 03 18:05:27.591175 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:27.591137    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:27.591279 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:27.591189    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"]
Nov 03 18:05:27.591279 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:27.591220    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:27.591279 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:27.591263    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:28.559754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:28.558895    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" event={"ID":"0459156f-933e-4bb4-88bc-976d5e41c36f","Type":"ContainerStarted","Data":"46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6"}
Nov 03 18:05:28.559754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:28.558994    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" event={"ID":"0459156f-933e-4bb4-88bc-976d5e41c36f","Type":"ContainerStarted","Data":"fe84a07c55d9c3977c0e1b85a73ebff0af80dfb9fa20dc3bb521b5515cb873d6"}
Nov 03 18:05:28.795679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:28.795618    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podStartSLOduration=2.795603168 podStartE2EDuration="2.795603168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:28.742351477 +0000 UTC m=+1183.972676552" watchObservedRunningTime="2026-11-03 18:05:28.795603168 +0000 UTC m=+1184.025928240"
Nov 03 18:05:29.166218 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:29.166180    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:29.169636 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:29.168342    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 18:05:29.169636 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:29.168398    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 18:05:30.165932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:30.165897    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 18:05:30.166320 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:30.165962    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 18:05:31.166202 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:31.166169    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 18:05:31.166603 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:31.166240    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 18:05:32.166245 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:32.166210    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 18:05:32.166713 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:32.166287    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 18:05:33.165860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:33.165815    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 18:05:33.166119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:33.165876    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 18:05:34.165597 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:34.165558    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 18:05:34.166103 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:34.165626    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 18:05:35.171560 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:35.171532    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:35.172038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:35.171962    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:35.172789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:35.172769    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:05:43.117005 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.116974    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"]
Nov 03 18:05:43.121704 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.121680    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.167506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.167480    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"]
Nov 03 18:05:43.171859 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.171839    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.181112 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.181091    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0adcbc-openshift-default-dockercfg-zctsw\""
Nov 03 18:05:43.222299 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.222268    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"]
Nov 03 18:05:43.234581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.234552    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"]
Nov 03 18:05:43.258061 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258031    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.258284 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258251    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258302    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.258419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258328    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258355    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.258419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258379    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.258419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258408    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258433    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258459    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258481    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258547    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgxh\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258620    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5h7k\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258664    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.258694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258692    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.259267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258814    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.259267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258869    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.259267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258936    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.259267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.258963    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360322 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360293    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360322 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360327    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360363    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360389    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360416    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360444    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360473    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360497    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360518    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360543    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360614    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgxh\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360694    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5h7k\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360754    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360787    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.360991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360796    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.360991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.360800    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.361392 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.361368    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.361559 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.361388    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.361644 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.361620    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.361709 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.361650    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.361794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.361702    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.361794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.361750    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.362072 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.362051    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.362253 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.362230    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.362632 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.362605    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.363623 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.363541    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.363954 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.363908    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.364054 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.363978    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.364139 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.364116    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.364666 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.364644    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.364938 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.364916    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.364938 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.364934    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.422970 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.422895    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.422970 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.422951    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgxh\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.423183 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.422986    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.423183 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.423126    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5h7k\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.436968 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.436943    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:43.486731 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.486685    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:43.860549 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:05:43.860512    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5272092_0dea_4ffd_93e1_e64256cffbd5.slice/crio-21207f6a226f2452c9ef84e961142e551309a260754036a8d4ff8e504bdde392 WatchSource:0}: Error finding container 21207f6a226f2452c9ef84e961142e551309a260754036a8d4ff8e504bdde392: Status 404 returned error can't find the container with id 21207f6a226f2452c9ef84e961142e551309a260754036a8d4ff8e504bdde392
Nov 03 18:05:43.863792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.863754    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:43.863909 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.863855    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"]
Nov 03 18:05:43.863909 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.863867    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:43.864023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.863914    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:43.910053 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:05:43.910024    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5501f821_1679_48b9_9bde_4dbaaf0450fb.slice/crio-473b6aa2183c9a9b8f248abfc6518ad4e9e14a8e8f8e4f094318568f35b082c7 WatchSource:0}: Error finding container 473b6aa2183c9a9b8f248abfc6518ad4e9e14a8e8f8e4f094318568f35b082c7: Status 404 returned error can't find the container with id 473b6aa2183c9a9b8f248abfc6518ad4e9e14a8e8f8e4f094318568f35b082c7
Nov 03 18:05:43.912794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.912765    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:43.912866 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.912832    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:43.912866 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.912857    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:05:43.913643 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:43.913624    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"]
Nov 03 18:05:44.714046 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.714006    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" event={"ID":"5501f821-1679-48b9-9bde-4dbaaf0450fb","Type":"ContainerStarted","Data":"0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73"}
Nov 03 18:05:44.714568 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.714547    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" event={"ID":"5501f821-1679-48b9-9bde-4dbaaf0450fb","Type":"ContainerStarted","Data":"473b6aa2183c9a9b8f248abfc6518ad4e9e14a8e8f8e4f094318568f35b082c7"}
Nov 03 18:05:44.716679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.716653    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" event={"ID":"d5272092-0dea-4ffd-93e1-e64256cffbd5","Type":"ContainerStarted","Data":"8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2"}
Nov 03 18:05:44.717574 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.717555    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" event={"ID":"d5272092-0dea-4ffd-93e1-e64256cffbd5","Type":"ContainerStarted","Data":"21207f6a226f2452c9ef84e961142e551309a260754036a8d4ff8e504bdde392"}
Nov 03 18:05:44.898400 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.898364    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:05:44.901964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.901906    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podStartSLOduration=1.901889217 podStartE2EDuration="1.901889217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:44.85453547 +0000 UTC m=+1200.084860543" watchObservedRunningTime="2026-11-03 18:05:44.901889217 +0000 UTC m=+1200.132214288"
Nov 03 18:05:44.987427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:44.987297    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podStartSLOduration=1.9872751690000001 podStartE2EDuration="1.987275169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:44.984357242 +0000 UTC m=+1200.214682316" watchObservedRunningTime="2026-11-03 18:05:44.987275169 +0000 UTC m=+1200.217600241"
Nov 03 18:05:45.194087 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.194060    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:05:45.196081 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.196059    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:05:45.443743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.439055    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:45.463745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.462095    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:45.463745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.462156    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:45.491773 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.491427    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:45.493744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.492679    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:45.495814 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:45.494300    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:46.446754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:46.440081    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:46.446754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:46.440141    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:46.496739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:46.494108    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:46.496739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:46.494168    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:47.452746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.444444    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:47.452746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.444520    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:47.487748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.487274    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:47.487748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.487354    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:47.815317 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.814804    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:47.851867 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.851825    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:47.871105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.871054    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:47.978741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.977821    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:47.997742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:47.997219    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.058668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.058621    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.303802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.272483    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:05:48.425339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.425106    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:05:48.425339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.425177    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.517345 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.496188    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:48.518898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.518857    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:48.519098 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.517098    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:48.519224 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.519201    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:48.608747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.602669    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:05:48.820144 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.820089    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:48.827745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.820805    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:05:48.940386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.877837    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:05:49.019744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:48.994636    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:49.283845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.139958    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:49.283845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.163885    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 18:05:49.283845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.180456    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:49.384743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.384404    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:49.406709 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.406678    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:05:49.407003 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.406989    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:05:49.407177 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.407164    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:05:49.407310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.407299    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:05:49.407446 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.407436    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:05:49.428182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.428153    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:05:49.457800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.457764    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:49.458102 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.458074    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:49.489104 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.489057    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:49.489408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.489370    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:49.516561 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.516520    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:05:49.526421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.526396    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:05:49.604877 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.604068    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:05:49.604877 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.604682    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:05:49.607285 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.606413    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:05:49.610288 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.610267    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:05:49.616742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.611192    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:05:49.627299 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:49.627274    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:05:50.479855 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:50.479811    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:50.480193 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:50.480158    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:50.487327 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:50.487297    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:50.487607 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:50.487562    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:51.438348 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:51.438307    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 18:05:51.438865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:51.438374    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 18:05:51.487128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:51.487086    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 18:05:51.487330 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:51.487157    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 18:05:52.796960 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:52.796788    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:52.797445 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:52.797424    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:52.798523 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:52.798481    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:05:52.805452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:52.805375    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:52.805999 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:52.805976    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:52.808123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:05:52.808103    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:05:54.522633 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:05:54.522597    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:06:12.215179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.215138    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"]
Nov 03 18:06:12.220040 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.220014    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.266487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.266452    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"]
Nov 03 18:06:12.270893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.270869    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.319255 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.319217    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"]
Nov 03 18:06:12.328125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328093    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328311 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328152    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsf6j\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328311 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328261    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxntv\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328309    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328421    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328474    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328619 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328593    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328640    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328676    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328767 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328714    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328809 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328800    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328843 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328832    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.328903 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328877    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328944    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.328994 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328974    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.329027 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.328997    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.329077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.329066    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.329114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.329094    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.363740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.343774    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"]
Nov 03 18:06:12.430374 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430338    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsf6j\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.430579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430384    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxntv\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.430579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430419    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.430579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430488    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.430579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430536    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430584    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430613    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430646    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430682    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430739    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430773    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.430871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430806    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.431530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430878    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.431530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430909    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.431530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.430933    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.431530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.431087    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.431530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.431055    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.431530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.431436    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.432035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.431841    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.432337    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.433376    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.433522    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.433621    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.433713    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.433770    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.434120    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.434129    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.434325    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.434454    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.434869    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.436080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.435144    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.436758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.436638    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.492956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.492890    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxntv\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.492956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.492894    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsf6j\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.493531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.493507    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.494161 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.494135    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.533914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.533880    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:12.573509 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.573476    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"]
Nov 03 18:06:12.580252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.580227    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.593059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.593030    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:12.650948 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.644650    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"]
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654139    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654187    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654216    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654269    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654296    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654358    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654418    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqgv\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654444    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.654841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.654490    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756540    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756592    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756619    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756672    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756704    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756789    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756861    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqgv\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756886    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.757020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.756936    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.758004 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.757980    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.758152 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.758060    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.758500 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.758481    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.759032 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.758985    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.759294 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.759269    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.761881 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.761847    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.762317 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.762283    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.819828 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.819796    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqgv\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.819968 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.819938    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.897196 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.897164    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:12.973905 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:06:12.973870    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09609ee_8a67_446d_90dc_6a841437ae6b.slice/crio-8f6b135e232b872342b6d103dfee0c38ccfff7c0ced0a2c581aafb87aa8e60c5 WatchSource:0}: Error finding container 8f6b135e232b872342b6d103dfee0c38ccfff7c0ced0a2c581aafb87aa8e60c5: Status 404 returned error can't find the container with id 8f6b135e232b872342b6d103dfee0c38ccfff7c0ced0a2c581aafb87aa8e60c5
Nov 03 18:06:12.988127 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.987081    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:12.988127 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.987173    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:12.988127 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.987228    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:12.991879 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:12.991816    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"]
Nov 03 18:06:13.032573 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.032546    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"]
Nov 03 18:06:13.037261 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:06:13.037228    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edfd3b8_2ccb_444f_913c_91bc502bc9a0.slice/crio-ddd00a21b4f15122ff077bd2ba4cd896b17bad4ed1aae884d5badf58f92cb8d0 WatchSource:0}: Error finding container ddd00a21b4f15122ff077bd2ba4cd896b17bad4ed1aae884d5badf58f92cb8d0: Status 404 returned error can't find the container with id ddd00a21b4f15122ff077bd2ba4cd896b17bad4ed1aae884d5badf58f92cb8d0
Nov 03 18:06:13.039581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.039542    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:13.040003 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.039973    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:13.040115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.040052    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:13.304987 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.304953    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"]
Nov 03 18:06:13.307686 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.307646    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:13.307833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.307765    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:13.307833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.307814    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:13.704927 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.704892    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" event={"ID":"a09609ee-8a67-446d-90dc-6a841437ae6b","Type":"ContainerStarted","Data":"0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c"}
Nov 03 18:06:13.705119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.704954    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" event={"ID":"a09609ee-8a67-446d-90dc-6a841437ae6b","Type":"ContainerStarted","Data":"8f6b135e232b872342b6d103dfee0c38ccfff7c0ced0a2c581aafb87aa8e60c5"}
Nov 03 18:06:13.706952 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.706920    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" event={"ID":"cb280431-6efb-4ee4-8446-ef91773809b3","Type":"ContainerStarted","Data":"5499979f58aeb86b4e632ea27a7e97e5bef2a4df39ae3c1dc8b2b620c4be22c8"}
Nov 03 18:06:13.707080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.706977    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" event={"ID":"cb280431-6efb-4ee4-8446-ef91773809b3","Type":"ContainerStarted","Data":"d728ff7fcf5481ac829443419d84e8cd5d651544e5b6714d3c396a66db549570"}
Nov 03 18:06:13.708769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.708744    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" event={"ID":"9edfd3b8-2ccb-444f-913c-91bc502bc9a0","Type":"ContainerStarted","Data":"12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab"}
Nov 03 18:06:13.708874 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.708797    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" event={"ID":"9edfd3b8-2ccb-444f-913c-91bc502bc9a0","Type":"ContainerStarted","Data":"ddd00a21b4f15122ff077bd2ba4cd896b17bad4ed1aae884d5badf58f92cb8d0"}
Nov 03 18:06:13.800028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.799986    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"]
Nov 03 18:06:13.804865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.804838    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.831517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.831464    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podStartSLOduration=1.831450437 podStartE2EDuration="1.831450437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.828297888 +0000 UTC m=+1229.058622960" watchObservedRunningTime="2026-11-03 18:06:13.831450437 +0000 UTC m=+1229.061775508"
Nov 03 18:06:13.863471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.863435    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"]
Nov 03 18:06:13.878043 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878003    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878181    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878298 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878286    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878330    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878384    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fch9m\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878417    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878458    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878494    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.878575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.878523    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.897373 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.897349    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:13.955266 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.955087    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podStartSLOduration=1.95506895 podStartE2EDuration="1.95506895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.949247934 +0000 UTC m=+1229.179573007" watchObservedRunningTime="2026-11-03 18:06:13.95506895 +0000 UTC m=+1229.185394023"
Nov 03 18:06:13.981680 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.981632    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.982432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.982407    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.983023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.982304    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.983286 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.983264    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.983791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.983768    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.983945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.983930    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.984814 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.984792    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fch9m\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.985257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.985238    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.985828 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.985809    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.986831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.986808    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.986942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.984684    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.986942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.986745    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.986942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.983641    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.987208 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.987188    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.998750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.988575    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:13.998750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:13.989080    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:14.066854 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.066781    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fch9m\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:14.089351 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.089315    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:14.120504 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.120473    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:14.223821 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.220364    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podStartSLOduration=2.220348419 podStartE2EDuration="2.220348419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:14.089872907 +0000 UTC m=+1229.320197980" watchObservedRunningTime="2026-11-03 18:06:14.220348419 +0000 UTC m=+1229.450673490"
Nov 03 18:06:14.554753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.552142    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:14.554753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.554468    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:14.554753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.554521    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:14.636513 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.595165    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:14.636513 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.622601    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:14.636513 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.622651    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:14.899013 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.898977    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:14.899301 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:14.899272    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:15.406276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.249699    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"]
Nov 03 18:06:15.406530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.360834    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:15.406777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.406743    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:15.407039 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.407017    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:15.592298 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.551769    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:15.706858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.706814    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:15.707607 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.564819    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:06:15.707914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.624973    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:15.708243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.708111    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:15.910793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.901054    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:15.910793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:15.901118    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:16.125739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.111945    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" event={"ID":"f40f64c2-53b5-4d1d-9f9d-b9207429f502","Type":"ContainerStarted","Data":"3c409ddb5b2e4cfa0e12a22301788acd4fb810eff0c8f68d7929e8a0488fb3ab"}
Nov 03 18:06:16.566742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.563735    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.566742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.563797    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:16.638354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.613449    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.638354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.613511    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:16.967199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.964823    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.967199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:16.964886    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:17.474744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.467487    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" event={"ID":"f40f64c2-53b5-4d1d-9f9d-b9207429f502","Type":"ContainerStarted","Data":"ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a"}
Nov 03 18:06:17.519755 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.517305    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.556743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.556568    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.581744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.560785    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:17.581744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.571799    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.606741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.603475    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.606741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.603535    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:17.718631 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.713121    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.760799 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.759813    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:17.786901 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.786857    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:06:17.794748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.792909    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podStartSLOduration=4.7928874740000005 podStartE2EDuration="4.792887474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:17.748290111 +0000 UTC m=+1232.978615173" watchObservedRunningTime="2026-11-03 18:06:17.792887474 +0000 UTC m=+1233.023212547"
Nov 03 18:06:17.800080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.799341    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:17.916262 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.914806    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:17.929756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.917259    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.929756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:17.917314    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:18.125136 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.125102    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:18.125451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.125430    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.125625 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.125604    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:18.717366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.717331    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.717668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.717624    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:18.880750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.750972    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.884835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.884793    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:18.885002 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.814858    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:06:18.888881 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.777980    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:18.920035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.889105    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:06:18.948743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.923828    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:06:18.969833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.931181    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.990741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:18.986816    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:19.123168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.123125    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.123463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.123434    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:19.438068 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.438007    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.569806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.537909    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.569806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.544328    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.569806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.544376    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:19.627180 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.627132    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.627469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.627431    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:19.727868 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.727639    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.873833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.747810    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.873833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.810967    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:19.912276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.897060    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.965745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.946875    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 18:06:19.984040 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.983947    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:19.991744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.987781    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:19.992027 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:19.987674    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.137449 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.137413    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.137749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.137702    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:20.162372 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.162334    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.444521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.444476    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.461593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.461565    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:06:20.548244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.548199    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.549829 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.549792    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:06:20.553142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.553121    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.568062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.567900    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:20.668558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.654838    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.668558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.654898    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:20.734797 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.734104    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:06:20.826341 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.826296    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:20.838742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.830136    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:06:20.860792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.838998    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:20.861353 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.839337    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:06:20.861845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.861823    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:06:20.862541 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.862523    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:06:20.913739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.903828    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:20.913739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.903881    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:20.948150 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:20.948104    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:21.016863 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.016758    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:21.150743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.130275    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:06:21.214773 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.167112    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.214773 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.211209    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:21.361601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.361565    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:06:21.362035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.362015    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:06:21.419749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.387314    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:06:21.473711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.471622    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:06:21.531753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.509407    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:06:21.531753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.516082    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:06:21.551786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.542328    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.551786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.542387    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 18:06:21.602350 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.602300    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.602600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.602567    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 18:06:21.604769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.604706    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:06:21.905743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.901821    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 18:06:21.905743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:21.901888    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 18:06:22.121337 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:22.121299    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:22.122034 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:22.121925    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:23.110517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.110454    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:23.113054 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.113032    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:23.114578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.114558    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:06:23.185116 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.185075    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:23.185617 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.185154    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:23.355348 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.355315    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:23.379203 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.379120    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:23.383195 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.383128    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:06:23.471177 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.471146    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:23.471867 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.471844    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:23.473731 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:23.473703    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:06:24.122030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:24.122000    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 18:06:24.122210 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:24.122048    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 18:06:24.542866 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:06:24.542767    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:06:25.549646 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:25.549614    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:25.551168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:25.551006    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:25.552975 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:25.552956    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:06:26.758948 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.758915    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"]
Nov 03 18:06:26.763249 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.763231    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.832476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.832422    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.832644 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.832493    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr589\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.832644 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.832580    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.832740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.832654    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.832801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.832777    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.832854 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.832839    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.833060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.833039    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.833115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.833099    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.833222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.833207    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.872655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.872620    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"]
Nov 03 18:06:26.934358 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934326    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934370    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934426    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934605 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934562    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934645 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934605    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr589\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934687 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934661    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934740    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934774    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934804    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934857    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.934942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.934889    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.935188 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.935140    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.935343 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.935322    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.935581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.935557    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.936700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.936679    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.937000 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.936980    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.998846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.998813    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:26.999023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:26.998920    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr589\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:27.081747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:27.077305    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:27.514223 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:06:27.514187    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaf5374_f308_4ee6_aa7c_05cebdc4d271.slice/crio-e7946c7592fa0db9aa9f3847d9a45bd3f5287ea306763c01f1e0efdbd5a963f2 WatchSource:0}: Error finding container e7946c7592fa0db9aa9f3847d9a45bd3f5287ea306763c01f1e0efdbd5a963f2: Status 404 returned error can't find the container with id e7946c7592fa0db9aa9f3847d9a45bd3f5287ea306763c01f1e0efdbd5a963f2
Nov 03 18:06:27.517197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:27.517160    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:27.517316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:27.517246    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:27.517393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:27.517345    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:27.517871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:27.517840    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"]
Nov 03 18:06:28.188744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:28.187667    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" event={"ID":"acaf5374-f308-4ee6-aa7c-05cebdc4d271","Type":"ContainerStarted","Data":"0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82"}
Nov 03 18:06:28.188744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:28.187757    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" event={"ID":"acaf5374-f308-4ee6-aa7c-05cebdc4d271","Type":"ContainerStarted","Data":"e7946c7592fa0db9aa9f3847d9a45bd3f5287ea306763c01f1e0efdbd5a963f2"}
Nov 03 18:06:28.389264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:28.389210    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podStartSLOduration=2.38919457 podStartE2EDuration="2.38919457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:28.386838106 +0000 UTC m=+1243.617163177" watchObservedRunningTime="2026-11-03 18:06:28.38919457 +0000 UTC m=+1243.619519642"
Nov 03 18:06:29.078685 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:29.078646    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:29.080257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:29.080230    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 18:06:29.080410 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:29.080286    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 18:06:30.078772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:30.078740    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 18:06:30.079169 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:30.078794    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 18:06:30.203817 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:30.203735    6167 generic.go:356] "Generic (PLEG): container finished" podID="57c37162-79cd-4a18-9346-87ccf834737f" containerID="d2173174b84414e92313c6d560a9434ab4c7e6ac771a9341439f66eecddea817" exitCode=0
Nov 03 18:06:30.203965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:30.203744    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2j58-qxzzg" event={"ID":"57c37162-79cd-4a18-9346-87ccf834737f","Type":"ContainerDied","Data":"d2173174b84414e92313c6d560a9434ab4c7e6ac771a9341439f66eecddea817"}
Nov 03 18:06:31.078386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.078349    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 18:06:31.078576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.078408    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 18:06:31.473137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.473103    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:06:31.565597 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.565562    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle\") pod \"57c37162-79cd-4a18-9346-87ccf834737f\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") "
Nov 03 18:06:31.565852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.565824    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle\") pod \"57c37162-79cd-4a18-9346-87ccf834737f\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") "
Nov 03 18:06:31.566019 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.565959    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq\" (UniqueName: \"kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq\") pod \"57c37162-79cd-4a18-9346-87ccf834737f\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") "
Nov 03 18:06:31.566019 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566000    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle" pod "57c37162-79cd-4a18-9346-87ccf834737f" (UID: "57c37162-79cd-4a18-9346-87ccf834737f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:31.566214 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566195    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path\") pod \"57c37162-79cd-4a18-9346-87ccf834737f\" (UID: \"57c37162-79cd-4a18-9346-87ccf834737f\") "
Nov 03 18:06:31.566351 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566324    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle" pod "57c37162-79cd-4a18-9346-87ccf834737f" (UID: "57c37162-79cd-4a18-9346-87ccf834737f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:31.566595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566576    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path" pod "57c37162-79cd-4a18-9346-87ccf834737f" (UID: "57c37162-79cd-4a18-9346-87ccf834737f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:31.566834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566805    6167 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/57c37162-79cd-4a18-9346-87ccf834737f-archives-path\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:31.566834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566830    6167 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-service-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:31.566966 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.566845    6167 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c37162-79cd-4a18-9346-87ccf834737f-trusted-ca-bundle\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:31.568911 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.568885    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq" pod "57c37162-79cd-4a18-9346-87ccf834737f" (UID: "57c37162-79cd-4a18-9346-87ccf834737f"). InnerVolumeSpecName "kube-api-access-qlfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:31.669209 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:31.669119    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlfkq\" (UniqueName: \"kubernetes.io/projected/57c37162-79cd-4a18-9346-87ccf834737f-kube-api-access-qlfkq\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:32.079760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.078541    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 18:06:32.079760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.078602    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 18:06:32.221428 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.221396    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2j58-qxzzg"
Nov 03 18:06:32.221565 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.221399    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2j58-qxzzg" event={"ID":"57c37162-79cd-4a18-9346-87ccf834737f","Type":"ContainerDied","Data":"232ef7bcfa0ae9d47a852d4b11aa8d81154383a2dfadac6eda794de0557dd54c"}
Nov 03 18:06:32.221565 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.221519    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232ef7bcfa0ae9d47a852d4b11aa8d81154383a2dfadac6eda794de0557dd54c"
Nov 03 18:06:32.715467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.715433    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7353-257ba6"]
Nov 03 18:06:32.716110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.716094    6167 cpu_manager.go:414] "containerMap: removing container" podUID="57c37162-79cd-4a18-9346-87ccf834737f" containerName="insights-gathering"
Nov 03 18:06:32.716151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.716111    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c37162-79cd-4a18-9346-87ccf834737f" containerName="insights-gathering"
Nov 03 18:06:32.716231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.716222    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c37162-79cd-4a18-9346-87ccf834737f" containerName="insights-gathering"
Nov 03 18:06:32.720564 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.720541    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:32.775250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.773688    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-257ba6"]
Nov 03 18:06:32.775758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.775692    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-447v9\""
Nov 03 18:06:32.888285 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.888251    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss7nf\" (UniqueName: \"kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf\") pod \"test-curl-7353-257ba6\" (UID: \"8e9b23a9-493a-4044-842b-8c1cbb74f930\") " pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:32.989988 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:32.989903    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss7nf\" (UniqueName: \"kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf\") pod \"test-curl-7353-257ba6\" (UID: \"8e9b23a9-493a-4044-842b-8c1cbb74f930\") " pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:33.066401 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:33.066366    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss7nf\" (UniqueName: \"kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf\") pod \"test-curl-7353-257ba6\" (UID: \"8e9b23a9-493a-4044-842b-8c1cbb74f930\") " pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:33.078137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:33.078108    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 18:06:33.078336 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:33.078316    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 18:06:33.366495 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:33.366452    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:33.780169 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:06:33.780133    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e9b23a9_493a_4044_842b_8c1cbb74f930.slice/crio-ed358550f1bdbd571519899702466eb8713e7f7c2ac9379059b8185474fe091b WatchSource:0}: Error finding container ed358550f1bdbd571519899702466eb8713e7f7c2ac9379059b8185474fe091b: Status 404 returned error can't find the container with id ed358550f1bdbd571519899702466eb8713e7f7c2ac9379059b8185474fe091b
Nov 03 18:06:33.782463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:33.782438    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:06:33.785615 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:33.785587    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7353-257ba6"]
Nov 03 18:06:34.078776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:34.078743    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 18:06:34.078951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:34.078807    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 18:06:34.238588 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:34.238546    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-257ba6" event={"ID":"8e9b23a9-493a-4044-842b-8c1cbb74f930","Type":"ContainerStarted","Data":"ed358550f1bdbd571519899702466eb8713e7f7c2ac9379059b8185474fe091b"}
Nov 03 18:06:35.083514 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:35.083475    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:35.084003 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:35.083967    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:35.084759 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:35.084734    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:06:36.255561 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:36.255522    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7353-257ba6" event={"ID":"8e9b23a9-493a-4044-842b-8c1cbb74f930","Type":"ContainerStarted","Data":"4edf8a0ac2e3eed4744223d8138f518eed45dde7b62dbd91cefb9e8461b84669"}
Nov 03 18:06:36.371060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:36.371012    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7353-257ba6" podStartSLOduration=2.848532081 podStartE2EDuration="4.370998576s" totalImagesPullingTime="1.522466495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:36.369056618 +0000 UTC m=+1251.599381689" watchObservedRunningTime="2026-11-03 18:06:36.370998576 +0000 UTC m=+1251.601323648"
Nov 03 18:06:38.141559 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:38.141523    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7353-257ba6"]
Nov 03 18:06:38.147416 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:38.147378    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7353-257ba6"]
Nov 03 18:06:38.270384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:38.270330    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7353-257ba6" podUID="8e9b23a9-493a-4044-842b-8c1cbb74f930" containerName="test-curl-7353-257ba6" containerID="cri-o://4edf8a0ac2e3eed4744223d8138f518eed45dde7b62dbd91cefb9e8461b84669" gracePeriod=2
Nov 03 18:06:38.325025 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:06:38.324969    6167 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7353-257ba6\" is forbidden: User \"system:node:ip-10-0-141-108.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-108.us-west-2.compute.internal' and this object" podUID="8e9b23a9-493a-4044-842b-8c1cbb74f930" pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:39.886338 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.886303    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"]
Nov 03 18:06:39.887123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.887104    6167 cpu_manager.go:414] "containerMap: removing container" podUID="8e9b23a9-493a-4044-842b-8c1cbb74f930" containerName="test-curl-7353-257ba6"
Nov 03 18:06:39.887179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.887124    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9b23a9-493a-4044-842b-8c1cbb74f930" containerName="test-curl-7353-257ba6"
Nov 03 18:06:39.887283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.887274    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9b23a9-493a-4044-842b-8c1cbb74f930" containerName="test-curl-7353-257ba6"
Nov 03 18:06:39.891065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.891040    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905034 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905007    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpqz\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905163 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905045    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905163 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905134    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905202    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905294 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905265    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905332 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905296    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905331    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905400 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905369    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.905441 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.905402    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:39.945518 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.945399    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-29bb4e29-openshift-default-dockercfg-c7vc8\""
Nov 03 18:06:39.999931 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:39.999893    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"]
Nov 03 18:06:40.007182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.007147    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.007322 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.007258    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.007732 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.007697    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.007851 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.007810    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.007924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.007910    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpqz\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.007979 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.007948    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.008052 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.008037    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.008138 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.008126    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.008256 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.008239    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.008323 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.008273    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.008588 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.008570    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.008676 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.008631    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.009253 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.009226    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.009383 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.009353    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.010391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.010367    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.010949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.010930    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.076020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.075457    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.076020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.075779    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpqz\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.215129 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.215033    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:40.300161 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.300129    6167 generic.go:356] "Generic (PLEG): container finished" podID="8e9b23a9-493a-4044-842b-8c1cbb74f930" containerID="4edf8a0ac2e3eed4744223d8138f518eed45dde7b62dbd91cefb9e8461b84669" exitCode=137
Nov 03 18:06:40.523465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.523440    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:40.621849 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.621815    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf\" (UniqueName: \"kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf\") pod \"8e9b23a9-493a-4044-842b-8c1cbb74f930\" (UID: \"8e9b23a9-493a-4044-842b-8c1cbb74f930\") "
Nov 03 18:06:40.623913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.623887    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf" pod "8e9b23a9-493a-4044-842b-8c1cbb74f930" (UID: "8e9b23a9-493a-4044-842b-8c1cbb74f930"). InnerVolumeSpecName "kube-api-access-ss7nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:40.678567 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:06:40.678535    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84743169_5dbc_428a_911c_d460225c12df.slice/crio-2c51d9a5a374538176e0cf5197cf647b762cfa9f1d12c7fae4fb71e1f38ca609 WatchSource:0}: Error finding container 2c51d9a5a374538176e0cf5197cf647b762cfa9f1d12c7fae4fb71e1f38ca609: Status 404 returned error can't find the container with id 2c51d9a5a374538176e0cf5197cf647b762cfa9f1d12c7fae4fb71e1f38ca609
Nov 03 18:06:40.681754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.681678    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:40.681852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.681779    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:40.681852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.681823    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:06:40.682393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.682375    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"]
Nov 03 18:06:40.724326 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.724302    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss7nf\" (UniqueName: \"kubernetes.io/projected/8e9b23a9-493a-4044-842b-8c1cbb74f930-kube-api-access-ss7nf\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:40.929390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:40.929361    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9b23a9-493a-4044-842b-8c1cbb74f930" path="/var/lib/kubelet/pods/8e9b23a9-493a-4044-842b-8c1cbb74f930/volumes"
Nov 03 18:06:41.308999 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:41.308909    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" event={"ID":"84743169-5dbc-428a-911c-d460225c12df","Type":"ContainerStarted","Data":"6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"}
Nov 03 18:06:41.308999 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:41.308970    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" event={"ID":"84743169-5dbc-428a-911c-d460225c12df","Type":"ContainerStarted","Data":"2c51d9a5a374538176e0cf5197cf647b762cfa9f1d12c7fae4fb71e1f38ca609"}
Nov 03 18:06:41.310268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:41.310236    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7353-257ba6"
Nov 03 18:06:41.310268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:41.310253    6167 scope.go:122] "RemoveContainer" containerID="4edf8a0ac2e3eed4744223d8138f518eed45dde7b62dbd91cefb9e8461b84669"
Nov 03 18:06:41.432641 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:41.432495    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podStartSLOduration=2.432478087 podStartE2EDuration="2.432478087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:41.430273926 +0000 UTC m=+1256.660599000" watchObservedRunningTime="2026-11-03 18:06:41.432478087 +0000 UTC m=+1256.662803160"
Nov 03 18:06:42.215578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:42.215546    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:42.221282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:42.221260    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:42.318990 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:42.318957    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:42.319854 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:42.319833    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:06:45.200625 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:45.200592    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:06:47.495861 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:47.495806    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:06:47.513872 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:47.513812    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.534742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:47.522795    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.744858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:47.744798    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.857912 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:47.857870    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:47.978865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:47.978814    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.232802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.232622    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:48.280742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.279911    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:06:48.280742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.280016    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.378061 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.378021    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:06:48.513804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.509836    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:06:48.531928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.515003    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:06:48.534688 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.534652    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:06:48.567292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.545545    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:06:48.567292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.560800    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.567292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.561185    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:06:48.891744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.888195    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:48.923806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.900484    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:06:48.969771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.966863    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:06:48.969771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:48.967422    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:06:49.051743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.048599    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.190802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.187014    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded"
Nov 03 18:06:49.263756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.239086    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.483800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.483002    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.552835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.517588    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.566689 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.566648    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:06:49.652758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.622751    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.768808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.760255    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:06:49.769095 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.769073    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:06:49.822238 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.822209    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:06:49.922755 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.922698    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:49.982260 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:49.972908    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.075609 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.075570    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.268202 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.268155    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.290280 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.289511    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:06:50.466779 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.466650    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:06:50.475482 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.475454    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:06:50.481623 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.481597    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:06:50.483956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.483897    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:06:50.521423 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.519742    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:06:50.541803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.536119    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:06:50.582616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.582588    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:06:50.584540 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.584520    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:06:50.603407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:50.603375    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:06:54.414873 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:54.414793    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"]
Nov 03 18:06:54.415300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:54.414979    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy" containerID="cri-o://e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d" gracePeriod=30
Nov 03 18:06:54.551381 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:06:54.551343    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:06:55.572909 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.572876    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"]
Nov 03 18:06:55.589483 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.583690    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.597164 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.597131    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"]
Nov 03 18:06:55.638799 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.638709    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-86101f5d\""
Nov 03 18:06:55.677616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.677573    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592x2\" (UniqueName: \"kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2\") pod \"payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") " pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.677872 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.677789    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") " pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.688360 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.688329    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-86101f5d-dockercfg-rfz8r\""
Nov 03 18:06:55.779435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.779388    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592x2\" (UniqueName: \"kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2\") pod \"payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") " pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.779699 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.779488    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") " pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.780278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.780254    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") " pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.841368 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.841297    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592x2\" (UniqueName: \"kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2\") pod \"payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") " pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:55.944306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:55.944270    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:56.322748 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:06:56.322701    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34de840e_a828_4625_9466_eb795c261e8d.slice/crio-0c9dcae03918549bf8a39d81c73838e682e42d78cf1bc0ee3d141f7d0286e9e6 WatchSource:0}: Error finding container 0c9dcae03918549bf8a39d81c73838e682e42d78cf1bc0ee3d141f7d0286e9e6: Status 404 returned error can't find the container with id 0c9dcae03918549bf8a39d81c73838e682e42d78cf1bc0ee3d141f7d0286e9e6
Nov 03 18:06:56.326415 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:56.326389    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"]
Nov 03 18:06:56.734484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:56.734394    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp" event={"ID":"34de840e-a828-4625-9466-eb795c261e8d","Type":"ContainerStarted","Data":"7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2"}
Nov 03 18:06:56.734484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:56.734454    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp" event={"ID":"34de840e-a828-4625-9466-eb795c261e8d","Type":"ContainerStarted","Data":"0c9dcae03918549bf8a39d81c73838e682e42d78cf1bc0ee3d141f7d0286e9e6"}
Nov 03 18:06:56.734928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:56.734627    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:06:56.853236 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:56.853172    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp" podStartSLOduration=1.853152665 podStartE2EDuration="1.853152665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:56.850787335 +0000 UTC m=+1272.081112404" watchObservedRunningTime="2026-11-03 18:06:56.853152665 +0000 UTC m=+1272.083477738"
Nov 03 18:06:59.700756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.700675    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:06:59.766294 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766265    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766486 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766378    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766486 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766423    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766486 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766445    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766650 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766539    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766650 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766619    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766647    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766729    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.766820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766759    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:59.766820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766792    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:59.767038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766853    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert\") pod \"3265c6d6-17a6-4705-9b2f-40407849d35c\" (UID: \"3265c6d6-17a6-4705-9b2f-40407849d35c\") "
Nov 03 18:06:59.767038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.766996    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:59.767038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767010    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:59.767420 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767394    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:59.767611 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767588    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.767704 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767616    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3265c6d6-17a6-4705-9b2f-40407849d35c-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.767704 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767630    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.767704 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767643    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.767704 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.767656    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.769215 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.769184    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:59.769330 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.769251    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:06:59.769395 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.769379    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "kube-api-access-ghhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:59.769475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.769455    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token" pod "3265c6d6-17a6-4705-9b2f-40407849d35c" (UID: "3265c6d6-17a6-4705-9b2f-40407849d35c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:59.772780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.772533    6167 generic.go:356] "Generic (PLEG): container finished" podID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerID="e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d" exitCode=0
Nov 03 18:06:59.772780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.772601    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"
Nov 03 18:06:59.772780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.772615    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" event={"ID":"3265c6d6-17a6-4705-9b2f-40407849d35c","Type":"ContainerDied","Data":"e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d"}
Nov 03 18:06:59.772780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.772707    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj" event={"ID":"3265c6d6-17a6-4705-9b2f-40407849d35c","Type":"ContainerDied","Data":"fec00bf2d070b0f5a464f6418a00bcae3caabd3a6f1bbc6a72f46607ba0be843"}
Nov 03 18:06:59.772780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.772762    6167 scope.go:122] "RemoveContainer" containerID="e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d"
Nov 03 18:06:59.817850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.817131    6167 scope.go:122] "RemoveContainer" containerID="e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d"
Nov 03 18:06:59.818491 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:06:59.818427    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d\": container with ID starting with e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d not found: ID does not exist" containerID="e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d"
Nov 03 18:06:59.818602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.818492    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d"} err="failed to get container status \"e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d\": rpc error: code = NotFound desc = could not find container \"e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d\": container with ID starting with e17bccce2d8184bf150c972fad5d386d03af758aee703149f83efd94f7c2485d not found: ID does not exist"
Nov 03 18:06:59.868771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.868741    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.868771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.868767    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhw8\" (UniqueName: \"kubernetes.io/projected/3265c6d6-17a6-4705-9b2f-40407849d35c-kube-api-access-ghhw8\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.868771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.868777    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:06:59.868771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:06:59.868787    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3265c6d6-17a6-4705-9b2f-40407849d35c-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:07:00.057878 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.057783    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"]
Nov 03 18:07:00.061269 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.061240    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6xhmj"]
Nov 03 18:07:00.797748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.788875    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"]
Nov 03 18:07:00.801191 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.799282    6167 cpu_manager.go:414] "containerMap: removing container" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy"
Nov 03 18:07:00.801402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.801382    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy"
Nov 03 18:07:00.801882 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.801862    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" containerName="istio-proxy"
Nov 03 18:07:00.814565 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.814533    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:00.844762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.841913    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"]
Nov 03 18:07:00.873124 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.873088    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-29bb4e29-dockercfg-s8kqs\""
Nov 03 18:07:00.873952 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.873924    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-29bb4e29\""
Nov 03 18:07:00.900918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.895840    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7h9\" (UniqueName: \"kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9\") pod \"payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") " pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:00.900918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.895929    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume\") pod \"payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") " pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:00.947826 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.947318    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3265c6d6-17a6-4705-9b2f-40407849d35c" path="/var/lib/kubelet/pods/3265c6d6-17a6-4705-9b2f-40407849d35c/volumes"
Nov 03 18:07:00.997555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.997515    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume\") pod \"payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") " pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:00.997910 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.997886    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7h9\" (UniqueName: \"kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9\") pod \"payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") " pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:00.998555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:00.998528    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume\") pod \"payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") " pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:01.061130 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.061052    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7h9\" (UniqueName: \"kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9\") pod \"payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") " pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:01.142772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.142708    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:01.522290 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:01.522236    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15e2565_465d_4159_907c_21a34e6aa937.slice/crio-7dc69e97293d562fc9244786b7bc2f5b7eab01b590a6df20a1e7c6593a0998eb WatchSource:0}: Error finding container 7dc69e97293d562fc9244786b7bc2f5b7eab01b590a6df20a1e7c6593a0998eb: Status 404 returned error can't find the container with id 7dc69e97293d562fc9244786b7bc2f5b7eab01b590a6df20a1e7c6593a0998eb
Nov 03 18:07:01.525110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.525087    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"]
Nov 03 18:07:01.805323 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.805230    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn" event={"ID":"b15e2565-465d-4159-907c-21a34e6aa937","Type":"ContainerStarted","Data":"46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc"}
Nov 03 18:07:01.805323 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.805287    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn" event={"ID":"b15e2565-465d-4159-907c-21a34e6aa937","Type":"ContainerStarted","Data":"7dc69e97293d562fc9244786b7bc2f5b7eab01b590a6df20a1e7c6593a0998eb"}
Nov 03 18:07:01.805874 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.805376    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:01.931676 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:01.931628    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn" podStartSLOduration=1.9316134969999998 podStartE2EDuration="1.931613497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:01.929263448 +0000 UTC m=+1277.159588520" watchObservedRunningTime="2026-11-03 18:07:01.931613497 +0000 UTC m=+1277.161938568"
Nov 03 18:07:04.790066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.790028    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"]
Nov 03 18:07:04.794412 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.794391    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.847936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.847899    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4b6e7259-openshift-default-dockercfg-tk9lj\""
Nov 03 18:07:04.857335 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.857293    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.859677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.859648    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.859829 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.859691    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.859881 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.859856    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.859982 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.859964    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.860068 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.860054    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.860124 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.860110    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.860204 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.860189    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxth\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.860261 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.860225    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:04.901987 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:04.901954    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"]
Nov 03 18:07:05.000925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.000845    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.000925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.000924    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001022    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxth\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001057    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001127    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001173    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001416 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001203    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001416 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001373    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.001539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.001518    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.028391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.022447    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.028391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.022794    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.028391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.023054    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.028391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.023277    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.028391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.024843    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.060173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.044141    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.060173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.057089    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.079741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.069443    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.125739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.122074    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxth\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth\") pod \"e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") " pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.408713 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.408670    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:05.836931 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:05.836902    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66465677_7842_4438_8e69_cbfbd6334744.slice/crio-4ba1ba07cb451c084be1a408382981396c2d460fed48708e0941a4671518156d WatchSource:0}: Error finding container 4ba1ba07cb451c084be1a408382981396c2d460fed48708e0941a4671518156d: Status 404 returned error can't find the container with id 4ba1ba07cb451c084be1a408382981396c2d460fed48708e0941a4671518156d
Nov 03 18:07:05.839520 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.839486    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:05.839594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.839554    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:05.839594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.839582    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:05.840901 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:05.840881    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"]
Nov 03 18:07:06.844681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:06.844585    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" event={"ID":"66465677-7842-4438-8e69-cbfbd6334744","Type":"ContainerStarted","Data":"55fca55b629e5839baaa6576d489d4eaaeaee12e5eb3fa101bf924154f0c237a"}
Nov 03 18:07:06.844681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:06.844674    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" event={"ID":"66465677-7842-4438-8e69-cbfbd6334744","Type":"ContainerStarted","Data":"4ba1ba07cb451c084be1a408382981396c2d460fed48708e0941a4671518156d"}
Nov 03 18:07:07.070917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:07.070857    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podStartSLOduration=3.070843268 podStartE2EDuration="3.070843268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:07.016928246 +0000 UTC m=+1282.247253318" watchObservedRunningTime="2026-11-03 18:07:07.070843268 +0000 UTC m=+1282.301168340"
Nov 03 18:07:07.409011 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:07.408978    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:07.414536 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:07.414512    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:07.745087 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:07.745001    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:07:07.851869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:07.851835    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:07.853287 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:07.853266    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:07:12.692077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.692046    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"]
Nov 03 18:07:12.698066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.698039    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749058 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749022    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749223 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749096    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749223 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749133    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749338 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749255    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749395 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749377    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxvk\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749449 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749410    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749519 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749502    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749566    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.749746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.749606    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.756205 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.756180    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"]
Nov 03 18:07:12.816219 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.816185    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:07:12.851304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851270    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.851464 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851383    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxvk\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.851464 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851420    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.851572 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851500    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851577    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851612    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851697    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851711    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.851813    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.852004    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852413 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.852073    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852413 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.852145    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852413 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.852182    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.852592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.852565    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.854489 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.854463    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.854613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.854602    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.913299 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.913273    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:12.913434 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:12.913341    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxvk\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:13.015259 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.015164    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:13.313824 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.313746    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"]
Nov 03 18:07:13.319913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.319887    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.345416 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.345385    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"]
Nov 03 18:07:13.349831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.349808    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.360043 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360006    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360175 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360066    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360238 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360175    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360330 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360314    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360389 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360342    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360442 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360384    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360541    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360597    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvssn\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.360672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.360648    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.374644 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.374617    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"]
Nov 03 18:07:13.438963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.438928    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"]
Nov 03 18:07:13.463017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.462988    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.463528 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463500    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvssn\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.463632 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463561    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mfw\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.463632 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463603    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.463763 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463668    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.463823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463803    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.463915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463860    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.463976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463912    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.463976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.463969    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.464082 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464014    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.464082 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464046    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.464178 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464140    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.464236 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464204    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.464236 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464230    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.464344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464254    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.464344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464290    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.464344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464318    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.464487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464356    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.464834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.464813    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.465711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.465681    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.465818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.465790    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.466494 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.466472    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.466850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.466816    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.469310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.468447    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:13.469310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.468529    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:13.469310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.468572    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:13.469589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.469318    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.471486 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.471464    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"]
Nov 03 18:07:13.471631 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.471579    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.526548 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.526522    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.526765 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.526548    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvssn\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.565306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565275    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565346    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565379    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565405    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565515    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mfw\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565592    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565650    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565737    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565842    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.565920 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.565879    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.566282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.566262    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.566415 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.566293    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.566934 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.566761    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.567057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.567034    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.569189 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.569161    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.569399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.569381    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.627652 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.627616    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.627858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.627713    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mfw\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.636643 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.636619    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:13.665999 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.665975    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:13.824063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.823977    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"]
Nov 03 18:07:13.833706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.833239    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.870888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.870856    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871045 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.870933    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871072    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871127    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871226 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871172    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871239    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xcz\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871273    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871811 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871547    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.871811 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.871634    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.892454 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.892425    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"]
Nov 03 18:07:13.923295 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.923256    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" event={"ID":"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c","Type":"ContainerStarted","Data":"b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f"}
Nov 03 18:07:13.923465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.923328    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" event={"ID":"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c","Type":"ContainerStarted","Data":"87e0858e0622ac75b6e549fa2717fd3da7d2c5d9c04988aa11923280068f308e"}
Nov 03 18:07:13.972921 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.972886    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.972953    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973004    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973082    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xcz\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973124    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973216    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973387 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973339    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973385    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973416    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973596 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973559    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973596 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973575    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.973785 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.973763    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.974184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.974158    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.974537 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.974513    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.976107 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.976082    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:13.976382 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:13.976365    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:14.016085 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.016057    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:14.017683 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.017662    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:14.017757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.017709    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:14.035556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.035529    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:14.035692 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.035631    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xcz\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:14.064368 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:14.064327    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3653f1be_c7ce_41bf_a5c8_1fdd40762aeb.slice/crio-cb6551511042eeaff1d09fdab40f225fe7bdf9ff9290bec6eedf1a44bd52ab13 WatchSource:0}: Error finding container cb6551511042eeaff1d09fdab40f225fe7bdf9ff9290bec6eedf1a44bd52ab13: Status 404 returned error can't find the container with id cb6551511042eeaff1d09fdab40f225fe7bdf9ff9290bec6eedf1a44bd52ab13
Nov 03 18:07:14.067232 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.067205    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"]
Nov 03 18:07:14.067877 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.067845    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:14.067940 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.067909    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:14.067976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.067939    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:14.079965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.079859    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podStartSLOduration=2.079843148 podStartE2EDuration="2.079843148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:14.079501687 +0000 UTC m=+1289.309826759" watchObservedRunningTime="2026-11-03 18:07:14.079843148 +0000 UTC m=+1289.310168224"
Nov 03 18:07:14.123932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.122405    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:14.123932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.122469    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:14.123932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.122494    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:14.123932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.123348    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"]
Nov 03 18:07:14.152053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:14.152018    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:15.412506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.412461    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:15.413138 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.413112    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:15.413283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.413265    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:15.494533 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.494493    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:15.578804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.520196    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:07:15.690745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.658158    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:15.829803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.659199    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:15.829803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.792569    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"]
Nov 03 18:07:15.829803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.822336    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" event={"ID":"3cd797ae-f45d-4466-ab72-fe20b4af183c","Type":"ContainerStarted","Data":"a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722"}
Nov 03 18:07:15.829803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.822389    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" event={"ID":"3cd797ae-f45d-4466-ab72-fe20b4af183c","Type":"ContainerStarted","Data":"47bbe47d78544adbd66acef0c8337a94494d9a79320cbb0378e51fed6a15d914"}
Nov 03 18:07:15.849747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.838677    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:15.986746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.961866    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:15.986746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.947447    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.025750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:15.933197    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.592919 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.592878    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:16.593508 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.593474    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:16.683745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.676130    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" event={"ID":"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb","Type":"ContainerStarted","Data":"cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f"}
Nov 03 18:07:16.683745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.676208    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" event={"ID":"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb","Type":"ContainerStarted","Data":"cb6551511042eeaff1d09fdab40f225fe7bdf9ff9290bec6eedf1a44bd52ab13"}
Nov 03 18:07:16.683745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.676241    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:16.683745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.678538    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:16.686216 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.686179    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.686655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.686626    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.687097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.687079    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:07:16.687263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.687251    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:07:16.687709 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.687689    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.722128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.690355    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.723104 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.690372    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:07:16.723458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.706500    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:16.723609 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.723586    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:16.723732 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.720005    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podStartSLOduration=3.7199832600000002 podStartE2EDuration="3.71998326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:16.679094114 +0000 UTC m=+1291.909419209" watchObservedRunningTime="2026-11-03 18:07:16.71998326 +0000 UTC m=+1291.950308334"
Nov 03 18:07:16.745263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.733834    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:16.745263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.733893    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:16.825275 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.824124    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:16.851157 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.826125    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:07:16.972744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.943955    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podStartSLOduration=3.943932046 podStartE2EDuration="3.943932046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:16.823828542 +0000 UTC m=+1292.054153616" watchObservedRunningTime="2026-11-03 18:07:16.943932046 +0000 UTC m=+1292.174257118"
Nov 03 18:07:17.154952 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:16.975260    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:07:17.155682 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.038185    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:07:17.210747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.206862    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:17.417216 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.417124    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:17.568748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.566263    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:07:17.568748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.566328    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:07:17.674803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.670856    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:17.674803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.670937    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:17.704742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.678852    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:17.704742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.678928    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:17.704994 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.704961    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.723742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.707388    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.724061 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.724024    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.749744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.731870    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:07:17.749744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.735275    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:17.749744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.736774    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 18:07:17.749744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.737961    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:07:17.749744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.738022    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:07:17.857740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:17.826389    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.135820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.135769    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:18.205747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.203118    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:18.205747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.202955    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.205747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.202903    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.619744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.594221    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" event={"ID":"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1","Type":"ContainerStarted","Data":"4364fcf1301a0140dd3a3e5c988160f971f5a7e8220f7fe3f78080607d53f145"}
Nov 03 18:07:18.723855 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.723819    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:18.724457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.724408    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:18.724682 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.724647    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.741870 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.741809    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.830775 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.830516    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:18.830775 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.830566    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:18.832875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.832840    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.833482 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.833348    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.833866 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.833839    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:07:18.834014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.833997    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:18.865751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.839988    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:07:18.884805 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:18.842883    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.072745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.047660    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:19.073055 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.073015    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:19.118659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.118613    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.216808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.207379    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:19.216808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.208144    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 18:07:19.743868 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.726168    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.775969    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.009475    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.847471    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.847068    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.887630    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.923594    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.020806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:19.932180    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.054758 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:20.047229    6167 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.108s"
Nov 03 18:07:20.131549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.131512    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:20.131773 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.131573    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:20.132215 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.132184    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:20.132339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.132234    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:20.178099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.166262    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.178099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.169007    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" event={"ID":"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1","Type":"ContainerStarted","Data":"866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf"}
Nov 03 18:07:20.223336 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.222441    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:07:20.224772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.223655    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:20.625162 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.625133    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:07:20.707749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.704140    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:20.707749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:20.704158    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.098780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.078752    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.402274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.114617    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:21.402510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.114919    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:21.402645 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.255070    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.407818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.255122    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.255159    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.255199    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.312109    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.328898    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.617510    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.386420    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.353916    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.648742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.618358    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj" podUID="4e1776c1-ff08-4072-bf71-83ed8d82beb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:21.757805 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.747839    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.757805 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.747904    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:21.757805 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.748437    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.757805 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.748485    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:21.781780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.781742    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:21.782090 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.782050    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:21.831741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.827342    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podStartSLOduration=8.827321488 podStartE2EDuration="8.827321488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:20.304922291 +0000 UTC m=+1295.535247364" watchObservedRunningTime="2026-11-03 18:07:21.827321488 +0000 UTC m=+1297.057646560"
Nov 03 18:07:21.831741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.830779    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:21.831741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:21.830931    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:07:22.005873 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.005809    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.075749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.075303    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.075749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.075368    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:22.124778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.120359    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:07:22.165743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.161170    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"]
Nov 03 18:07:22.171739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.168792    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.172038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.172010    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.172304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.172276    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:22.184508 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.183761    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.315110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.315074    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-86101f5d-serving-cert\""
Nov 03 18:07:22.343794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.331525    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxfh\" (UniqueName: \"kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.344363 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.344336    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.369794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.367162    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.370315 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.370288    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.373320 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.339992    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"]
Nov 03 18:07:22.403212 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.394597    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:07:22.408842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.408802    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.473811 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.473771    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.475583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.475555    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.477806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.477783    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxfh\" (UniqueName: \"kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.478293 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.478272    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.543743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.532445    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.584837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.584799    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxfh\" (UniqueName: \"kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.633739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.633627    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.647796    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.728916    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.647876    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.733378    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.688183    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.733461    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:22.741760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.647836    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:22.888805 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:22.859863    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs\") pod \"maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:23.155791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.155032    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:23.282746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.206107    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.282746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.206165    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:23.282746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.213481    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.282746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.241086    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.282746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.241155    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:23.374743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.374179    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:07:23.431261 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.431202    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:23.533615 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.533540    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Nov 03 18:07:23.656229 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.656185    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.666176 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.666087    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.666418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.666384    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:23.667318 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.667295    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:23.667480 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.667460    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:23.809742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.732535    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.830757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.743829    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:23.870769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:23.866773    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.194776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.170795    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.195310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.194797    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.241110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.223526    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.241110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.223638    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:24.266097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.264488    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.314730 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.297500    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.369742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.323250    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.369742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.324110    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:24.431905 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.431847    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.678744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.676868    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:24.678744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.678498    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:07:24.787742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.769531    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:24.787742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.769610    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:24.851750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.834384    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:07:24.885746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.862892    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:24.952233 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:24.920433    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.009284 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.009219    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.009711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.009564    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:25.389952 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.330541    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.658252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.594884    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:25.658252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.407925    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.658252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.341264    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.658252 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:25.341467    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d616aa3_295c_4b7b_a5df_25351fe937f4.slice/crio-cf6a922bbebebaf800cb7d4f663a63485d49a44f6501d11b9ec883da8a72eeca WatchSource:0}: Error finding container cf6a922bbebebaf800cb7d4f663a63485d49a44f6501d11b9ec883da8a72eeca: Status 404 returned error can't find the container with id cf6a922bbebebaf800cb7d4f663a63485d49a44f6501d11b9ec883da8a72eeca
Nov 03 18:07:25.658252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.417254    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.781681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.704830    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.781681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.750050    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.805983 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.783878    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 18:07:25.805983 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.783939    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8" podUID="3fe9fc29-00d4-46f5-ab5b-df8dd964a65c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.805983 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.785906    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:25.805983 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.785959    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:25.860861 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.860807    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:25.961966 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:25.961870    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.115196 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.115134    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:26.411751 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:26.391954    6167 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.454s"
Nov 03 18:07:26.411751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.392071    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:07:26.411751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.392175    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"]
Nov 03 18:07:26.411751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.392266    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:07:26.646222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.646162    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss" event={"ID":"2d616aa3-295c-4b7b-a5df-25351fe937f4","Type":"ContainerStarted","Data":"cf6a922bbebebaf800cb7d4f663a63485d49a44f6501d11b9ec883da8a72eeca"}
Nov 03 18:07:26.661134 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.661099    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.661446 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.661398    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:26.762642 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:26.762539    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:07:26.762927 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.762896    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.791944 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.791889    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.792206 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.792174    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:26.797295 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.797255    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.848993 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.848521    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.849281 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.849237    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 18:07:26.877791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.877749    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 18:07:26.879393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.879359    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 18:07:26.880772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.880702    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:26.907552 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.907509    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:26.950021 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.940902    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:26.980044 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:26.980006    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.099234 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.046131    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 18:07:27.136842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.047637    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.196650 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.123414    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.198375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.136829    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 18:07:27.199771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.199736    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:27.199948 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.199922    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:27.473375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.473341    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:07:27.699819 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.699379    6167 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-m7nkv" podUID="6f01ae27-6219-4cf1-a0b7-5e1f190c2887" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Nov 03 18:07:27.857918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.857877    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:27.945793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:27.927042    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:07:28.004059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.001855    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:28.004059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.003194    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:28.004059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.003237    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:28.016238 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.016209    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:28.092603 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.092543    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:28.139818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.139237    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:28.170149 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.144499    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 18:07:28.216957 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.209478    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:07:28.216957 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.209535    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:07:28.241930 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.219107    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:28.241930 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.219162    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:28.243368 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.243045    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 18:07:28.298902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.298873    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:28.444479 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.444332    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:07:28.486888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.484708    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:07:28.490378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.488408    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:07:28.494355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.494329    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:07:28.497497 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.497475    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:07:28.571679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.571653    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:07:28.598163 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.598132    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:07:28.642747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.639260    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:28.665496 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.665369    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:28.667192 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.667156    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 18:07:28.667383 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.667359    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 18:07:28.667605 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.667590    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:28.847745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.845469    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss" event={"ID":"2d616aa3-295c-4b7b-a5df-25351fe937f4","Type":"ContainerStarted","Data":"9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0"}
Nov 03 18:07:28.847745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.845527    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:28.847745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.846852    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:07:28.850553 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:28.850485    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:07:29.184491 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:29.184399    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 18:07:29.184789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:29.184755    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 18:07:29.188933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:29.188907    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:29.981743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:29.981694    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:29.984773 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:29.984679    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:07:30.127749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.123350    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss" podStartSLOduration=8.663514367 podStartE2EDuration="10.123329546s" totalImagesPullingTime="1.459815179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:30.11899998 +0000 UTC m=+1305.349325052" watchObservedRunningTime="2026-11-03 18:07:30.123329546 +0000 UTC m=+1305.353654618"
Nov 03 18:07:30.425259 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.425221    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"]
Nov 03 18:07:30.435784 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.435255    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.491247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.490784    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-29bb4e29-serving-cert\""
Nov 03 18:07:30.493621 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.493591    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"]
Nov 03 18:07:30.500462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.500305    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2wn\" (UniqueName: \"kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.500462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.500408    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.500462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.500447    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.500753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.500537    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.540787 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.538561    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:30.560334 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.560303    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:07:30.615321 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.615281    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2wn\" (UniqueName: \"kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.615505 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.615395    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.615505 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.615432    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.615619 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.615533    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.617614 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.617583    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.617765 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:30.617683    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-serving-cert: secret "maas-api-e2e-ait-29bb4e29-serving-cert" not found
Nov 03 18:07:30.617765 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:30.617760    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls podName:48c735b7-f479-4448-80e9-4876898a61b2 nodeName:}" failed. No retries permitted until 2026-11-03 18:07:31.117741686 +0000 UTC m=+1306.348066740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls") pod "maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" (UID: "48c735b7-f479-4448-80e9-4876898a61b2") : secret "maas-api-e2e-ait-29bb4e29-serving-cert" not found
Nov 03 18:07:30.622576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.622546    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:30.681929 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:30.681832    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2wn\" (UniqueName: \"kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:31.127170 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:31.127136    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:31.129944 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:31.129912    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\") pod \"maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:31.388613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:31.388531    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:31.775802 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:31.775710    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c735b7_f479_4448_80e9_4876898a61b2.slice/crio-515140a1d9230c38fc8dd1862a61ce88a8247165a3cda0c79b3a9211a2434c4f WatchSource:0}: Error finding container 515140a1d9230c38fc8dd1862a61ce88a8247165a3cda0c79b3a9211a2434c4f: Status 404 returned error can't find the container with id 515140a1d9230c38fc8dd1862a61ce88a8247165a3cda0c79b3a9211a2434c4f
Nov 03 18:07:31.779124 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:31.779063    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"]
Nov 03 18:07:31.889741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:31.887099    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" event={"ID":"48c735b7-f479-4448-80e9-4876898a61b2","Type":"ContainerStarted","Data":"515140a1d9230c38fc8dd1862a61ce88a8247165a3cda0c79b3a9211a2434c4f"}
Nov 03 18:07:32.899292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:32.899255    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" event={"ID":"48c735b7-f479-4448-80e9-4876898a61b2","Type":"ContainerStarted","Data":"9eebce5a833f4e197b7d9da485c13809141d8e5b9ed4bf683c3f3860698ff9dc"}
Nov 03 18:07:32.899769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:32.899472    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:33.021471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:33.021364    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" podStartSLOduration=2.108139868 podStartE2EDuration="3.021344851s" totalImagesPullingTime="913.204983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:33.018498985 +0000 UTC m=+1308.248824057" watchObservedRunningTime="2026-11-03 18:07:33.021344851 +0000 UTC m=+1308.251669925"
Nov 03 18:07:34.868835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:34.868799    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:07:35.061791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:35.059171    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:07:38.912246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:38.912216    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:07:42.579137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.579094    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"]
Nov 03 18:07:42.585317 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.585293    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726312 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726273    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726336    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726525 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726505    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726590 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726553    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726613    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726685 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726640    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726779 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726758    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726814    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.726902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.726883    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pww\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.794862 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.794827    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"]
Nov 03 18:07:42.829043 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829007    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829240 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829155    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829240 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829191    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pww\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829489 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829464    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829584 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829487    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829584 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829507    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829584 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829530    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829631    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829669    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.829865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.829693    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.830064 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.830036    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.830231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.830086    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.830420 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.830394    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.830562 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.830537    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.835382 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.835359    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.835672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.835648    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.892106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.892072    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.892260 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.892159    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pww\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:42.900791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:42.900765    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:43.329373 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:43.329338    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2716032b_1df8_4066_bff3_9ee3e2898296.slice/crio-5936e4b55e46001d5eb7c8963b114614b5ad32d5c7740dc736b7147e8aec38b9 WatchSource:0}: Error finding container 5936e4b55e46001d5eb7c8963b114614b5ad32d5c7740dc736b7147e8aec38b9: Status 404 returned error can't find the container with id 5936e4b55e46001d5eb7c8963b114614b5ad32d5c7740dc736b7147e8aec38b9
Nov 03 18:07:43.332207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:43.332172    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:43.332309 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:43.332239    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:43.332309 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:43.332269    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:07:43.332972 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:43.332947    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"]
Nov 03 18:07:44.008284 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:44.008246    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" event={"ID":"2716032b-1df8-4066-bff3-9ee3e2898296","Type":"ContainerStarted","Data":"1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5"}
Nov 03 18:07:44.008804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:44.008305    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" event={"ID":"2716032b-1df8-4066-bff3-9ee3e2898296","Type":"ContainerStarted","Data":"5936e4b55e46001d5eb7c8963b114614b5ad32d5c7740dc736b7147e8aec38b9"}
Nov 03 18:07:44.216482 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:44.216431    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podStartSLOduration=2.216417522 podStartE2EDuration="2.216417522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:44.179860942 +0000 UTC m=+1319.410186014" watchObservedRunningTime="2026-11-03 18:07:44.216417522 +0000 UTC m=+1319.446742652"
Nov 03 18:07:44.901877 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:44.901838    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:44.903706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:44.903673    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:07:44.903853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:44.903729    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 18:07:45.901293 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:45.901261    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:07:45.901687 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:45.901313    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 18:07:46.901551 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:46.901511    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:07:46.901984 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:46.901565    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 18:07:47.901480 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:47.901441    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:07:47.901661 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:47.901494    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 18:07:48.901676 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:48.901639    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:07:48.902108 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:48.901700    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 18:07:49.902072 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:49.902042    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 18:07:49.902458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:49.902093    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 18:07:50.906428 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:50.906398    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:50.906952 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:50.906888    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:50.907480 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:50.907459    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:07:53.888841 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.886115    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"]
Nov 03 18:07:53.896138 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.896111    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:53.946859 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.946822    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"]
Nov 03 18:07:53.951325 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.951295    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-248f8ee3-serving-cert\""
Nov 03 18:07:53.957911 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.957879    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:53.958095 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.958066    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:53.958292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.958271    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:53.958372 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:53.958329    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tzd\" (UniqueName: \"kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.059738 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.059688    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.059928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.059814    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tzd\" (UniqueName: \"kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.059928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.059906    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.060125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.060110    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.060304 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:54.060289    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-serving-cert: secret "maas-api-e2e-ait-248f8ee3-serving-cert" not found
Nov 03 18:07:54.060370 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:54.060360    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls podName:a013347c-d9fc-465a-88aa-79c47c760bb5 nodeName:}" failed. No retries permitted until 2026-11-03 18:07:54.560341184 +0000 UTC m=+1329.790666251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls") pod "maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" (UID: "a013347c-d9fc-465a-88aa-79c47c760bb5") : secret "maas-api-e2e-ait-248f8ee3-serving-cert" not found
Nov 03 18:07:54.060607 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.060582    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.063235 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.063209    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.124013 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.123984    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tzd\" (UniqueName: \"kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.522538 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:07:54.522501    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:07:54.574435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.574399    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.577107 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.577078    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\") pod \"maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:54.816538 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:54.816511    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:55.203381 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:07:55.203350    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda013347c_d9fc_465a_88aa_79c47c760bb5.slice/crio-4369c92025ae4d60f004c2ba201628c0b41038a45126fff8a5c8b96450b29c25 WatchSource:0}: Error finding container 4369c92025ae4d60f004c2ba201628c0b41038a45126fff8a5c8b96450b29c25: Status 404 returned error can't find the container with id 4369c92025ae4d60f004c2ba201628c0b41038a45126fff8a5c8b96450b29c25
Nov 03 18:07:55.207602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:55.207577    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"]
Nov 03 18:07:56.129653 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:56.129626    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" event={"ID":"a013347c-d9fc-465a-88aa-79c47c760bb5","Type":"ContainerStarted","Data":"4369c92025ae4d60f004c2ba201628c0b41038a45126fff8a5c8b96450b29c25"}
Nov 03 18:07:57.138871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:57.138830    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" event={"ID":"a013347c-d9fc-465a-88aa-79c47c760bb5","Type":"ContainerStarted","Data":"a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2"}
Nov 03 18:07:57.139361 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:57.139334    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:07:57.261176 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:07:57.261122    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" podStartSLOduration=3.40706855 podStartE2EDuration="4.261107187s" totalImagesPullingTime="854.038637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:57.258288545 +0000 UTC m=+1332.488613620" watchObservedRunningTime="2026-11-03 18:07:57.261107187 +0000 UTC m=+1332.491432258"
Nov 03 18:08:01.787785 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.786229    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"]
Nov 03 18:08:01.817741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.813131    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.886744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.874808    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"]
Nov 03 18:08:01.886947 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.886812    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.886947 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.886867    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.886947 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.886938    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjk8\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.887151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.886989    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.887151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.887024    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.887151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.887124    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.887314 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.887247    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:01.941744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.917774    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 18:08:02.087790 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.936089    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 18:08:02.111602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.936180    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xxxzp\""
Nov 03 18:08:02.111860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:01.936201    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 18:08:02.111957 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.014969    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.112207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.112187    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.206745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.190409    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.206745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.190644    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.206745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.190684    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.206745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.190790    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjk8\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.206745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.190851    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.206745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.023907    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.222009 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.118325    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.247867 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.222954    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.248123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.229098    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.251804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.229098    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.299684 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.277238    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.321580 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.321543    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjk8\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5pnsv\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.495784 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.495673    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:02.962560 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:02.962521    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"]
Nov 03 18:08:02.985199 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:08:02.985170    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a1f178_4833_4db8_95a3_d597ec75369a.slice/crio-ad20a9e500c8bcbe5626829a68dc7f68683bee5fedf5f9708ffa1bbc4cadc4ca WatchSource:0}: Error finding container ad20a9e500c8bcbe5626829a68dc7f68683bee5fedf5f9708ffa1bbc4cadc4ca: Status 404 returned error can't find the container with id ad20a9e500c8bcbe5626829a68dc7f68683bee5fedf5f9708ffa1bbc4cadc4ca
Nov 03 18:08:03.160840 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:03.160809    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:08:03.426458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:03.426398    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" event={"ID":"86a1f178-4833-4db8-95a3-d597ec75369a","Type":"ContainerStarted","Data":"ad20a9e500c8bcbe5626829a68dc7f68683bee5fedf5f9708ffa1bbc4cadc4ca"}
Nov 03 18:08:07.938860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:07.938823    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:07.939274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:07.938904    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:08.502782 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:08.502746    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" event={"ID":"86a1f178-4833-4db8-95a3-d597ec75369a","Type":"ContainerStarted","Data":"bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863"}
Nov 03 18:08:08.503000 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:08.502981    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:08.680949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:08.680893    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" podStartSLOduration=2.730677448 podStartE2EDuration="7.680875928s" totalImagesPullingTime="4.95019848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:08.675878544 +0000 UTC m=+1343.906203640" watchObservedRunningTime="2026-11-03 18:08:08.680875928 +0000 UTC m=+1343.911200997"
Nov 03 18:08:09.526616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:09.526583    6167 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5pnsv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:08:09.527197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:09.527167    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" podUID="86a1f178-4833-4db8-95a3-d597ec75369a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:08:11.542695 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:11.542606    6167 generic.go:356] "Generic (PLEG): container finished" podID="84743169-5dbc-428a-911c-d460225c12df" containerID="6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4" exitCode=0
Nov 03 18:08:11.542695 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:11.542664    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" event={"ID":"84743169-5dbc-428a-911c-d460225c12df","Type":"ContainerDied","Data":"6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"}
Nov 03 18:08:11.543206 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:11.543188    6167 scope.go:122] "RemoveContainer" containerID="6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"
Nov 03 18:08:11.544958 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:11.544925    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:11.545088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:11.545007    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:11.545088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:11.545057    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:12.319362 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:12.319334    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:08:12.523563 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:12.523533    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:08:12.578621 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:12.578524    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" event={"ID":"84743169-5dbc-428a-911c-d460225c12df","Type":"ContainerStarted","Data":"bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67"}
Nov 03 18:08:13.215296 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:13.215266    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:08:13.216445 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:13.216405    6167 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 18:08:13.216575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:13.216463    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 18:08:14.216200 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:14.216162    6167 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 18:08:14.216597 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:14.216222    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 18:08:15.215760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:15.215709    6167 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 18:08:15.216025 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:15.215781    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 18:08:16.215888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:16.215847    6167 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 18:08:16.216305 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:16.215912    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 18:08:17.216218 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:17.216182    6167 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 18:08:17.216766 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:17.216238    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 18:08:18.220800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:18.220761    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:08:18.221252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:18.221240    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:08:18.222016 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:18.221994    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:08:24.522128 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:08:24.522043    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:08:25.580928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:25.580888    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:08:28.768989 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:28.767809    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:28.838083 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:28.837840    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:28.880693 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:28.879984    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:29.020797 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:29.018834    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:29.036243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:29.033798    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:29.036243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:29.034178    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 18:08:29.222889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:29.222829    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:29.332795 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:29.326361    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:08:33.523122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:33.523037    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"]
Nov 03 18:08:37.853887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:37.853841    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:37.854377 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:37.853920    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:38.373585 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.373550    6167 generic.go:356] "Generic (PLEG): container finished" podID="66465677-7842-4438-8e69-cbfbd6334744" containerID="55fca55b629e5839baaa6576d489d4eaaeaee12e5eb3fa101bf924154f0c237a" exitCode=0
Nov 03 18:08:38.373790 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.373629    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" event={"ID":"66465677-7842-4438-8e69-cbfbd6334744","Type":"ContainerDied","Data":"55fca55b629e5839baaa6576d489d4eaaeaee12e5eb3fa101bf924154f0c237a"}
Nov 03 18:08:38.374212 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.374195    6167 scope.go:122] "RemoveContainer" containerID="55fca55b629e5839baaa6576d489d4eaaeaee12e5eb3fa101bf924154f0c237a"
Nov 03 18:08:38.376042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.376016    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:38.376114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.376088    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:38.376150 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.376120    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:38.523696 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.523642    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp" podUID="34de840e-a828-4625-9466-eb795c261e8d" containerName="payload-processing" containerID="cri-o://7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2" gracePeriod=25
Nov 03 18:08:38.895528 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:38.895461    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:08:39.021934 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.021905    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume\") pod \"34de840e-a828-4625-9466-eb795c261e8d\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") "
Nov 03 18:08:39.022212 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.022194    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2\" (UniqueName: \"kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2\") pod \"34de840e-a828-4625-9466-eb795c261e8d\" (UID: \"34de840e-a828-4625-9466-eb795c261e8d\") "
Nov 03 18:08:39.022313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.022287    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume" pod "34de840e-a828-4625-9466-eb795c261e8d" (UID: "34de840e-a828-4625-9466-eb795c261e8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:39.022937 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.022887    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34de840e-a828-4625-9466-eb795c261e8d-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:39.024580 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.024552    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2" pod "34de840e-a828-4625-9466-eb795c261e8d" (UID: "34de840e-a828-4625-9466-eb795c261e8d"). InnerVolumeSpecName "kube-api-access-592x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:39.124632 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.124590    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-592x2\" (UniqueName: \"kubernetes.io/projected/34de840e-a828-4625-9466-eb795c261e8d-kube-api-access-592x2\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:39.384373 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.384334    6167 generic.go:356] "Generic (PLEG): container finished" podID="34de840e-a828-4625-9466-eb795c261e8d" containerID="7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2" exitCode=0
Nov 03 18:08:39.384553 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.384529    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp" event={"ID":"34de840e-a828-4625-9466-eb795c261e8d","Type":"ContainerDied","Data":"7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2"}
Nov 03 18:08:39.384657 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.384569    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp" event={"ID":"34de840e-a828-4625-9466-eb795c261e8d","Type":"ContainerDied","Data":"0c9dcae03918549bf8a39d81c73838e682e42d78cf1bc0ee3d141f7d0286e9e6"}
Nov 03 18:08:39.384657 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.384588    6167 scope.go:122] "RemoveContainer" containerID="7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2"
Nov 03 18:08:39.384888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.384702    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"
Nov 03 18:08:39.389354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.389244    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" event={"ID":"66465677-7842-4438-8e69-cbfbd6334744","Type":"ContainerStarted","Data":"28512e13b9531920549f08bdfc559b8eb1fea007173d7259f4f87b19612656de"}
Nov 03 18:08:39.399898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.399874    6167 scope.go:122] "RemoveContainer" containerID="7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2"
Nov 03 18:08:39.400207 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:08:39.400180    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2\": container with ID starting with 7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2 not found: ID does not exist" containerID="7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2"
Nov 03 18:08:39.400287 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.400217    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2"} err="failed to get container status \"7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2\": rpc error: code = NotFound desc = could not find container \"7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2\": container with ID starting with 7eb1346a3772e735cb0b264ab0996bea9e95f2b9acb7759a3c5f66ca582e17b2 not found: ID does not exist"
Nov 03 18:08:39.409555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.409522    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:08:39.409808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.409773    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:39.409945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.409823    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:39.683591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.683496    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"]
Nov 03 18:08:39.692784 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:39.692740    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-86101f5d-6ffd4f4b9f-npdpp"]
Nov 03 18:08:40.409387 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:40.409349    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:40.409794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:40.409431    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:40.936744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:40.930968    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34de840e-a828-4625-9466-eb795c261e8d" path="/var/lib/kubelet/pods/34de840e-a828-4625-9466-eb795c261e8d/volumes"
Nov 03 18:08:41.409263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:41.409227    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:41.409437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:41.409283    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:42.410111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:42.410078    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:42.410560 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:42.410128    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:43.409220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:43.409181    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:43.409404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:43.409240    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:44.409082 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:44.409045    6167 patch_prober.go:29] interesting pod/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 18:08:44.409474 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:44.409102    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 18:08:45.414098 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:45.414067    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:08:45.414525 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:45.414502    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:08:45.415183 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:45.415164    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:08:48.658682 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:48.658630    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"]
Nov 03 18:08:53.660106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:53.660057    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn" podUID="b15e2565-465d-4159-907c-21a34e6aa937" containerName="payload-processing" containerID="cri-o://46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc" gracePeriod=25
Nov 03 18:08:53.981352 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:53.981327    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:08:54.053384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.053349    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9\" (UniqueName: \"kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9\") pod \"b15e2565-465d-4159-907c-21a34e6aa937\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") "
Nov 03 18:08:54.053584 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.053394    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume\") pod \"b15e2565-465d-4159-907c-21a34e6aa937\" (UID: \"b15e2565-465d-4159-907c-21a34e6aa937\") "
Nov 03 18:08:54.053913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.053888    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume" pod "b15e2565-465d-4159-907c-21a34e6aa937" (UID: "b15e2565-465d-4159-907c-21a34e6aa937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:54.054182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.054165    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15e2565-465d-4159-907c-21a34e6aa937-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:54.055662 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.055633    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9" pod "b15e2565-465d-4159-907c-21a34e6aa937" (UID: "b15e2565-465d-4159-907c-21a34e6aa937"). InnerVolumeSpecName "kube-api-access-wv7h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:54.158151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.158110    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7h9\" (UniqueName: \"kubernetes.io/projected/b15e2565-465d-4159-907c-21a34e6aa937-kube-api-access-wv7h9\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:08:54.555744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.555380    6167 generic.go:356] "Generic (PLEG): container finished" podID="b15e2565-465d-4159-907c-21a34e6aa937" containerID="46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc" exitCode=0
Nov 03 18:08:54.555744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.555511    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn" event={"ID":"b15e2565-465d-4159-907c-21a34e6aa937","Type":"ContainerDied","Data":"46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc"}
Nov 03 18:08:54.555744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.555562    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn" event={"ID":"b15e2565-465d-4159-907c-21a34e6aa937","Type":"ContainerDied","Data":"7dc69e97293d562fc9244786b7bc2f5b7eab01b590a6df20a1e7c6593a0998eb"}
Nov 03 18:08:54.555744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.555591    6167 scope.go:122] "RemoveContainer" containerID="46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc"
Nov 03 18:08:54.556130 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.555765    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"
Nov 03 18:08:54.584302 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:08:54.584146    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:08:54.586306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.586276    6167 scope.go:122] "RemoveContainer" containerID="46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc"
Nov 03 18:08:54.586835 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:08:54.586670    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc\": container with ID starting with 46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc not found: ID does not exist" containerID="46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc"
Nov 03 18:08:54.586835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.586710    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc"} err="failed to get container status \"46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc\": rpc error: code = NotFound desc = could not find container \"46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc\": container with ID starting with 46944a2434a9d0a49091356bd3edc9985374682c403e77b370376550e119f7fc not found: ID does not exist"
Nov 03 18:08:54.732849 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.732787    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"]
Nov 03 18:08:54.736474 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.736451    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29bb4e29-b9db8595c-vgtwn"]
Nov 03 18:08:54.941482 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:54.941445    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15e2565-465d-4159-907c-21a34e6aa937" path="/var/lib/kubelet/pods/b15e2565-465d-4159-907c-21a34e6aa937/volumes"
Nov 03 18:08:55.385131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.385100    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"]
Nov 03 18:08:55.385946 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.385928    6167 cpu_manager.go:414] "containerMap: removing container" podUID="b15e2565-465d-4159-907c-21a34e6aa937" containerName="payload-processing"
Nov 03 18:08:55.386024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.385947    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15e2565-465d-4159-907c-21a34e6aa937" containerName="payload-processing"
Nov 03 18:08:55.386024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.385963    6167 cpu_manager.go:414] "containerMap: removing container" podUID="34de840e-a828-4625-9466-eb795c261e8d" containerName="payload-processing"
Nov 03 18:08:55.386024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.385968    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de840e-a828-4625-9466-eb795c261e8d" containerName="payload-processing"
Nov 03 18:08:55.386125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.386085    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de840e-a828-4625-9466-eb795c261e8d" containerName="payload-processing"
Nov 03 18:08:55.386125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.386104    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15e2565-465d-4159-907c-21a34e6aa937" containerName="payload-processing"
Nov 03 18:08:55.391839 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.391817    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490298 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490263    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490485 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490417    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490486    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490545    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490692 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490572    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490692 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490597    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490692 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490659    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjktk\" (UniqueName: \"kubernetes.io/projected/eabe656f-c8dd-4a46-9e7c-53694a6748ca-kube-api-access-rjktk\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490699    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.490898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.490832    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.510654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.510620    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"]
Nov 03 18:08:55.592584 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592535    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.592584 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592594    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.592860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592769    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.592928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592875    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjktk\" (UniqueName: \"kubernetes.io/projected/eabe656f-c8dd-4a46-9e7c-53694a6748ca-kube-api-access-rjktk\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.592977 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592926    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593032 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592968    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593032 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.592997    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593044    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593201 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593182    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593299 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593282    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593327    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593379 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593365    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593481    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.593980 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.593957    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.595561 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.595539    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.595828 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.595809    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.654251 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.654181    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eabe656f-c8dd-4a46-9e7c-53694a6748ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.654435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.654403    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjktk\" (UniqueName: \"kubernetes.io/projected/eabe656f-c8dd-4a46-9e7c-53694a6748ca-kube-api-access-rjktk\") pod \"maas-default-gateway-openshift-default-6dcf757899-97fx2\" (UID: \"eabe656f-c8dd-4a46-9e7c-53694a6748ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:55.707470 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:55.707435    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:56.133232 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:08:56.133191    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabe656f_c8dd_4a46_9e7c_53694a6748ca.slice/crio-17fb0b8a8717967c63f3f83665403751aea79b68609f044956297ef77d460407 WatchSource:0}: Error finding container 17fb0b8a8717967c63f3f83665403751aea79b68609f044956297ef77d460407: Status 404 returned error can't find the container with id 17fb0b8a8717967c63f3f83665403751aea79b68609f044956297ef77d460407
Nov 03 18:08:56.136369 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.136333    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:56.136477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.136406    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:56.136477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.136433    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:08:56.136998 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.136932    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"]
Nov 03 18:08:56.586296 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.586262    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" event={"ID":"eabe656f-c8dd-4a46-9e7c-53694a6748ca","Type":"ContainerStarted","Data":"8e1e2c86e1b77c12d3a7ad0269f4eda769fb5b5edea80ecfbd8334e3d0f0e2ce"}
Nov 03 18:08:56.586455 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.586318    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" event={"ID":"eabe656f-c8dd-4a46-9e7c-53694a6748ca","Type":"ContainerStarted","Data":"17fb0b8a8717967c63f3f83665403751aea79b68609f044956297ef77d460407"}
Nov 03 18:08:56.707575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.707536    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:08:56.709706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:56.709657    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podStartSLOduration=1.7096454570000001 podStartE2EDuration="1.709645457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:56.707485947 +0000 UTC m=+1391.937811022" watchObservedRunningTime="2026-11-03 18:08:56.709645457 +0000 UTC m=+1391.939970529"
Nov 03 18:08:57.709644 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:57.709606    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:08:57.710046 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:57.709656    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:08:58.708588 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:58.708550    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:08:58.708798 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:58.708614    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:08:59.708217 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:59.708183    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:08:59.708751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:08:59.708238    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:09:00.708810 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:00.708762    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:09:00.709233 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:00.708817    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:09:01.708030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:01.707943    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:09:01.708030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:01.708009    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:09:02.708093 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:02.708059    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:09:02.708481 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:02.708113    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:09:03.708631 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:03.708587    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 18:09:03.709063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:03.708664    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 18:09:05.423125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:05.423077    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:09:05.423869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:05.423847    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:09:05.424754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:05.424708    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2"
Nov 03 18:09:14.277956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.277916    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"]
Nov 03 18:09:14.278454 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.278141    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss" podUID="2d616aa3-295c-4b7b-a5df-25351fe937f4" containerName="maas-api" containerID="cri-o://9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0" gracePeriod=30
Nov 03 18:09:14.611852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.611823    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:09:14.806908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.801357    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls\") pod \"2d616aa3-295c-4b7b-a5df-25351fe937f4\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") "
Nov 03 18:09:14.806908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.801447    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh\" (UniqueName: \"kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh\") pod \"2d616aa3-295c-4b7b-a5df-25351fe937f4\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") "
Nov 03 18:09:14.806908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.801701    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs\") pod \"2d616aa3-295c-4b7b-a5df-25351fe937f4\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") "
Nov 03 18:09:14.806908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.801782    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca\") pod \"2d616aa3-295c-4b7b-a5df-25351fe937f4\" (UID: \"2d616aa3-295c-4b7b-a5df-25351fe937f4\") "
Nov 03 18:09:14.819447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.819392    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh" pod "2d616aa3-295c-4b7b-a5df-25351fe937f4" (UID: "2d616aa3-295c-4b7b-a5df-25351fe937f4"). InnerVolumeSpecName "kube-api-access-rfxfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:14.822902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.822120    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca" pod "2d616aa3-295c-4b7b-a5df-25351fe937f4" (UID: "2d616aa3-295c-4b7b-a5df-25351fe937f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:14.830880 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.830852    6167 generic.go:356] "Generic (PLEG): container finished" podID="2d616aa3-295c-4b7b-a5df-25351fe937f4" containerID="9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0" exitCode=0
Nov 03 18:09:14.831025 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.830934    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"
Nov 03 18:09:14.831142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.830971    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss" event={"ID":"2d616aa3-295c-4b7b-a5df-25351fe937f4","Type":"ContainerDied","Data":"9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0"}
Nov 03 18:09:14.831230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.831186    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss" event={"ID":"2d616aa3-295c-4b7b-a5df-25351fe937f4","Type":"ContainerDied","Data":"cf6a922bbebebaf800cb7d4f663a63485d49a44f6501d11b9ec883da8a72eeca"}
Nov 03 18:09:14.831230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.831213    6167 scope.go:122] "RemoveContainer" containerID="9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0"
Nov 03 18:09:14.834233 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.834193    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs" pod "2d616aa3-295c-4b7b-a5df-25351fe937f4" (UID: "2d616aa3-295c-4b7b-a5df-25351fe937f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:14.834761 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.834705    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls" pod "2d616aa3-295c-4b7b-a5df-25351fe937f4" (UID: "2d616aa3-295c-4b7b-a5df-25351fe937f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:14.866452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.866392    6167 scope.go:122] "RemoveContainer" containerID="9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0"
Nov 03 18:09:14.867058 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:09:14.867007    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0\": container with ID starting with 9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0 not found: ID does not exist" containerID="9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0"
Nov 03 18:09:14.867165 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.867069    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0"} err="failed to get container status \"9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0\": rpc error: code = NotFound desc = could not find container \"9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0\": container with ID starting with 9b2ecb2c8557f61eb783bbfd0dd385bffa0319ac3c8d0ef68eb7aeeea59961c0 not found: ID does not exist"
Nov 03 18:09:14.904487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.904458    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:14.904487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.904487    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d616aa3-295c-4b7b-a5df-25351fe937f4-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:14.904652 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.904496    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d616aa3-295c-4b7b-a5df-25351fe937f4-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:14.904652 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:14.904505    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxfh\" (UniqueName: \"kubernetes.io/projected/2d616aa3-295c-4b7b-a5df-25351fe937f4-kube-api-access-rfxfh\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:15.256337 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:15.256116    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"]
Nov 03 18:09:15.306383 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:15.306339    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-86101f5d-78cc654747-gh6ss"]
Nov 03 18:09:16.946926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:16.946885    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d616aa3-295c-4b7b-a5df-25351fe937f4" path="/var/lib/kubelet/pods/2d616aa3-295c-4b7b-a5df-25351fe937f4/volumes"
Nov 03 18:09:20.445137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:20.445085    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"]
Nov 03 18:09:20.445635 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:20.445354    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" podUID="48c735b7-f479-4448-80e9-4876898a61b2" containerName="maas-api" containerID="cri-o://9eebce5a833f4e197b7d9da485c13809141d8e5b9ed4bf683c3f3860698ff9dc" gracePeriod=30
Nov 03 18:09:21.000812 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.000767    6167 generic.go:356] "Generic (PLEG): container finished" podID="48c735b7-f479-4448-80e9-4876898a61b2" containerID="9eebce5a833f4e197b7d9da485c13809141d8e5b9ed4bf683c3f3860698ff9dc" exitCode=0
Nov 03 18:09:21.005593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.005547    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" event={"ID":"48c735b7-f479-4448-80e9-4876898a61b2","Type":"ContainerDied","Data":"9eebce5a833f4e197b7d9da485c13809141d8e5b9ed4bf683c3f3860698ff9dc"}
Nov 03 18:09:21.005777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.005621    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc" event={"ID":"48c735b7-f479-4448-80e9-4876898a61b2","Type":"ContainerDied","Data":"515140a1d9230c38fc8dd1862a61ce88a8247165a3cda0c79b3a9211a2434c4f"}
Nov 03 18:09:21.005777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.005645    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="515140a1d9230c38fc8dd1862a61ce88a8247165a3cda0c79b3a9211a2434c4f"
Nov 03 18:09:21.007124 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.007095    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:09:21.163817 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.145446    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn\" (UniqueName: \"kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn\") pod \"48c735b7-f479-4448-80e9-4876898a61b2\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") "
Nov 03 18:09:21.164059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.164028    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca\") pod \"48c735b7-f479-4448-80e9-4876898a61b2\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") "
Nov 03 18:09:21.164214 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.164155    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\") pod \"48c735b7-f479-4448-80e9-4876898a61b2\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") "
Nov 03 18:09:21.164394 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.164376    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs\") pod \"48c735b7-f479-4448-80e9-4876898a61b2\" (UID: \"48c735b7-f479-4448-80e9-4876898a61b2\") "
Nov 03 18:09:21.165509 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.157843    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn" pod "48c735b7-f479-4448-80e9-4876898a61b2" (UID: "48c735b7-f479-4448-80e9-4876898a61b2"). InnerVolumeSpecName "kube-api-access-lt2wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:21.166070 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.166046    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca" pod "48c735b7-f479-4448-80e9-4876898a61b2" (UID: "48c735b7-f479-4448-80e9-4876898a61b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:21.167588 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.167563    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs" pod "48c735b7-f479-4448-80e9-4876898a61b2" (UID: "48c735b7-f479-4448-80e9-4876898a61b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:21.168705 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.168675    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls" pod "48c735b7-f479-4448-80e9-4876898a61b2" (UID: "48c735b7-f479-4448-80e9-4876898a61b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:21.267079 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.267038    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c735b7-f479-4448-80e9-4876898a61b2-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:21.267079 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.267077    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:21.267282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.267091    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c735b7-f479-4448-80e9-4876898a61b2-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:21.267282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:21.267105    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2wn\" (UniqueName: \"kubernetes.io/projected/48c735b7-f479-4448-80e9-4876898a61b2-kube-api-access-lt2wn\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:22.009777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:22.009702    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"
Nov 03 18:09:22.179981 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:22.179948    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"]
Nov 03 18:09:22.232086 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:22.232049    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-29bb4e29-bd89dcf8-v6pfc"]
Nov 03 18:09:22.931388 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:22.931352    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c735b7-f479-4448-80e9-4876898a61b2" path="/var/lib/kubelet/pods/48c735b7-f479-4448-80e9-4876898a61b2/volumes"
Nov 03 18:09:23.608963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:23.608925    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"]
Nov 03 18:09:23.609452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:23.609203    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy" containerID="cri-o://bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67" gracePeriod=30
Nov 03 18:09:24.524036 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:09:24.523995    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:09:28.913596 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.913567    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:09:28.992166 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992131    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992166 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992174    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992264    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992300    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992437    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992479    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992457    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.992655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992549    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.992655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992581    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.992655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992597    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:28.992655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992638    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992990 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992788    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.992990 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.992894    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert\") pod \"84743169-5dbc-428a-911c-d460225c12df\" (UID: \"84743169-5dbc-428a-911c-d460225c12df\") "
Nov 03 18:09:28.993362 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.993291    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:28.993996 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.993972    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.994097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.993999    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.994097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.994013    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.994097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.994027    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.994097 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.994041    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84743169-5dbc-428a-911c-d460225c12df-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:28.994869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.994841    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:28.994982 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.994877    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "kube-api-access-mhpqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:28.994982 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.994903    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:28.995066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:28.995038    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy" pod "84743169-5dbc-428a-911c-d460225c12df" (UID: "84743169-5dbc-428a-911c-d460225c12df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:29.075687 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.075652    6167 generic.go:356] "Generic (PLEG): container finished" podID="84743169-5dbc-428a-911c-d460225c12df" containerID="bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67" exitCode=0
Nov 03 18:09:29.075893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.075775    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"
Nov 03 18:09:29.075893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.075780    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" event={"ID":"84743169-5dbc-428a-911c-d460225c12df","Type":"ContainerDied","Data":"bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67"}
Nov 03 18:09:29.075893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.075877    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn" event={"ID":"84743169-5dbc-428a-911c-d460225c12df","Type":"ContainerDied","Data":"2c51d9a5a374538176e0cf5197cf647b762cfa9f1d12c7fae4fb71e1f38ca609"}
Nov 03 18:09:29.076003 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.075909    6167 scope.go:122] "RemoveContainer" containerID="bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67"
Nov 03 18:09:29.092983 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.088679    6167 scope.go:122] "RemoveContainer" containerID="6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"
Nov 03 18:09:29.096045 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.096016    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.096184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.096052    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpqz\" (UniqueName: \"kubernetes.io/projected/84743169-5dbc-428a-911c-d460225c12df-kube-api-access-mhpqz\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.096184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.096067    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84743169-5dbc-428a-911c-d460225c12df-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.096184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.096079    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84743169-5dbc-428a-911c-d460225c12df-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:29.102837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.102811    6167 scope.go:122] "RemoveContainer" containerID="bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67"
Nov 03 18:09:29.103215 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:09:29.103188    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67\": container with ID starting with bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67 not found: ID does not exist" containerID="bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67"
Nov 03 18:09:29.103324 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.103228    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67"} err="failed to get container status \"bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67\": rpc error: code = NotFound desc = could not find container \"bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67\": container with ID starting with bd8c826de49be135237a44c69ee87af94d48d72bd3b00a7e91c2f8a1e0955b67 not found: ID does not exist"
Nov 03 18:09:29.103324 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.103253    6167 scope.go:122] "RemoveContainer" containerID="6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"
Nov 03 18:09:29.103565 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:09:29.103547    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4\": container with ID starting with 6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4 not found: ID does not exist" containerID="6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"
Nov 03 18:09:29.103629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.103571    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4"} err="failed to get container status \"6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4\": rpc error: code = NotFound desc = could not find container \"6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4\": container with ID starting with 6037db8d68480b309e9554075920592a3f6f227f118ef322d693e6dccc28dab4 not found: ID does not exist"
Nov 03 18:09:29.262771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.262669    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"]
Nov 03 18:09:29.267977 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:29.267948    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6gktn"]
Nov 03 18:09:30.930891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:30.930857    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84743169-5dbc-428a-911c-d460225c12df" path="/var/lib/kubelet/pods/84743169-5dbc-428a-911c-d460225c12df/volumes"
Nov 03 18:09:35.392014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:35.391976    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:09:40.330264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.330226    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"]
Nov 03 18:09:40.330783 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.330431    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" podUID="a013347c-d9fc-465a-88aa-79c47c760bb5" containerName="maas-api" containerID="cri-o://a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2" gracePeriod=30
Nov 03 18:09:40.717476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.717182    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:09:40.731972 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.731935    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"]
Nov 03 18:09:40.733105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733084    6167 cpu_manager.go:414] "containerMap: removing container" podUID="48c735b7-f479-4448-80e9-4876898a61b2" containerName="maas-api"
Nov 03 18:09:40.733198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733107    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c735b7-f479-4448-80e9-4876898a61b2" containerName="maas-api"
Nov 03 18:09:40.733198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733144    6167 cpu_manager.go:414] "containerMap: removing container" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy"
Nov 03 18:09:40.733198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733153    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy"
Nov 03 18:09:40.733198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733184    6167 cpu_manager.go:414] "containerMap: removing container" podUID="a013347c-d9fc-465a-88aa-79c47c760bb5" containerName="maas-api"
Nov 03 18:09:40.733198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733194    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a013347c-d9fc-465a-88aa-79c47c760bb5" containerName="maas-api"
Nov 03 18:09:40.733384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733233    6167 cpu_manager.go:414] "containerMap: removing container" podUID="2d616aa3-295c-4b7b-a5df-25351fe937f4" containerName="maas-api"
Nov 03 18:09:40.733384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733240    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d616aa3-295c-4b7b-a5df-25351fe937f4" containerName="maas-api"
Nov 03 18:09:40.733457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733387    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="a013347c-d9fc-465a-88aa-79c47c760bb5" containerName="maas-api"
Nov 03 18:09:40.733457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733401    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy"
Nov 03 18:09:40.733457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733415    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d616aa3-295c-4b7b-a5df-25351fe937f4" containerName="maas-api"
Nov 03 18:09:40.733457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733428    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c735b7-f479-4448-80e9-4876898a61b2" containerName="maas-api"
Nov 03 18:09:40.733457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.733442    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy"
Nov 03 18:09:40.741023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.740982    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.818060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.815172    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"]
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.831349    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd\" (UniqueName: \"kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd\") pod \"a013347c-d9fc-465a-88aa-79c47c760bb5\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") "
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.831459    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca\") pod \"a013347c-d9fc-465a-88aa-79c47c760bb5\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") "
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.831508    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs\") pod \"a013347c-d9fc-465a-88aa-79c47c760bb5\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") "
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.831896    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\") pod \"a013347c-d9fc-465a-88aa-79c47c760bb5\" (UID: \"a013347c-d9fc-465a-88aa-79c47c760bb5\") "
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832262    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7f9\" (UniqueName: \"kubernetes.io/projected/63fe4ab5-a21e-4224-bc27-57fc19495f6e-kube-api-access-9v7f9\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832329    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832377    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832474    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.832542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832505    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.833223 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832562    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.833223 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.832646    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.852765 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.835474    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.852765 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.835589    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.852765 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.848988    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca" pod "a013347c-d9fc-465a-88aa-79c47c760bb5" (UID: "a013347c-d9fc-465a-88aa-79c47c760bb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:40.864743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.859016    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd" pod "a013347c-d9fc-465a-88aa-79c47c760bb5" (UID: "a013347c-d9fc-465a-88aa-79c47c760bb5"). InnerVolumeSpecName "kube-api-access-q8tzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:40.879740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.872691    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls" pod "a013347c-d9fc-465a-88aa-79c47c760bb5" (UID: "a013347c-d9fc-465a-88aa-79c47c760bb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:40.879740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.873524    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs" pod "a013347c-d9fc-465a-88aa-79c47c760bb5" (UID: "a013347c-d9fc-465a-88aa-79c47c760bb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939094    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939201    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7f9\" (UniqueName: \"kubernetes.io/projected/63fe4ab5-a21e-4224-bc27-57fc19495f6e-kube-api-access-9v7f9\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939252    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939300    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939415    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939445    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939499    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939579    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939780    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939873    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939890    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tzd\" (UniqueName: \"kubernetes.io/projected/a013347c-d9fc-465a-88aa-79c47c760bb5-kube-api-access-q8tzd\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939906    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a013347c-d9fc-465a-88aa-79c47c760bb5-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.939921    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a013347c-d9fc-465a-88aa-79c47c760bb5-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:40.941463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.940773    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.947710 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.947683    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.948039 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.948012    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.948264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.948243    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.948783 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.948484    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.951803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.951757    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:40.954113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:40.954092    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:41.001923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.001888    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4ab5-a21e-4224-bc27-57fc19495f6e-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:41.002344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.002309    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7f9\" (UniqueName: \"kubernetes.io/projected/63fe4ab5-a21e-4224-bc27-57fc19495f6e-kube-api-access-9v7f9\") pod \"maas-default-gateway-openshift-default-6dcf757899-qp8zz\" (UID: \"63fe4ab5-a21e-4224-bc27-57fc19495f6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:41.095417 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.095379    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:41.191283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.190705    6167 generic.go:356] "Generic (PLEG): container finished" podID="a013347c-d9fc-465a-88aa-79c47c760bb5" containerID="a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2" exitCode=0
Nov 03 18:09:41.191283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.190975    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" event={"ID":"a013347c-d9fc-465a-88aa-79c47c760bb5","Type":"ContainerDied","Data":"a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2"}
Nov 03 18:09:41.191283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.191045    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp" event={"ID":"a013347c-d9fc-465a-88aa-79c47c760bb5","Type":"ContainerDied","Data":"4369c92025ae4d60f004c2ba201628c0b41038a45126fff8a5c8b96450b29c25"}
Nov 03 18:09:41.191283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.191067    6167 scope.go:122] "RemoveContainer" containerID="a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2"
Nov 03 18:09:41.191283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.191194    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"
Nov 03 18:09:41.209209 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.209102    6167 scope.go:122] "RemoveContainer" containerID="a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2"
Nov 03 18:09:41.209470 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:09:41.209445    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2\": container with ID starting with a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2 not found: ID does not exist" containerID="a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2"
Nov 03 18:09:41.209552 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.209478    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2"} err="failed to get container status \"a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2\": rpc error: code = NotFound desc = could not find container \"a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2\": container with ID starting with a520f770e6895e8db35c8f1402676f7cb1607c0394aadc7e9043727188f1d8a2 not found: ID does not exist"
Nov 03 18:09:41.498895 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.498796    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"]
Nov 03 18:09:41.506399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.506021    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-248f8ee3-5d466b7474-ngmtp"]
Nov 03 18:09:41.573620 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:09:41.573587    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fe4ab5_a21e_4224_bc27_57fc19495f6e.slice/crio-8f72ff65a61f8e933661cb62ed79ecbd7478d566b9e470f14668c69f7189bf6e WatchSource:0}: Error finding container 8f72ff65a61f8e933661cb62ed79ecbd7478d566b9e470f14668c69f7189bf6e: Status 404 returned error can't find the container with id 8f72ff65a61f8e933661cb62ed79ecbd7478d566b9e470f14668c69f7189bf6e
Nov 03 18:09:41.576385 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.576351    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:09:41.576500 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.576417    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:09:41.576500 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.576444    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:09:41.577553 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:41.577334    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"]
Nov 03 18:09:42.201465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:42.201431    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" event={"ID":"63fe4ab5-a21e-4224-bc27-57fc19495f6e","Type":"ContainerStarted","Data":"b3f1fadd1aa15e7b4bb58f3e69bd903ecdd838fa652f1b6b7573d73b1c21d94c"}
Nov 03 18:09:42.201630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:42.201484    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" event={"ID":"63fe4ab5-a21e-4224-bc27-57fc19495f6e","Type":"ContainerStarted","Data":"8f72ff65a61f8e933661cb62ed79ecbd7478d566b9e470f14668c69f7189bf6e"}
Nov 03 18:09:42.330865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:42.330801    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podStartSLOduration=2.330782912 podStartE2EDuration="2.330782912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:42.325700254 +0000 UTC m=+1437.556025326" watchObservedRunningTime="2026-11-03 18:09:42.330782912 +0000 UTC m=+1437.561107982"
Nov 03 18:09:42.949742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:42.948741    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a013347c-d9fc-465a-88aa-79c47c760bb5" path="/var/lib/kubelet/pods/a013347c-d9fc-465a-88aa-79c47c760bb5/volumes"
Nov 03 18:09:43.095894 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:43.095858    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:43.097820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:43.097789    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:43.097993 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:43.097847    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:44.096477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:44.096443    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:44.096889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:44.096498    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:45.096262 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:45.096230    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:45.096436 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:45.096280    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:46.096618 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:46.096578    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:46.097022 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:46.096646    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:47.096103 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:47.096066    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:47.096398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:47.096366    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:48.096054 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:48.096019    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:48.096476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:48.096073    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:49.102758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:49.099219    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 18:09:49.102758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:49.099321    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 18:09:49.102758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:49.099693    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"]
Nov 03 18:09:49.102758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:49.101050    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy" containerID="cri-o://28512e13b9531920549f08bdfc559b8eb1fea007173d7259f4f87b19612656de" gracePeriod=30
Nov 03 18:09:51.126603 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:51.126565    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:09:51.127235 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:51.127145    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:09:51.150021 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:51.149980    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:51.150671 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:51.150642    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:51.152354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:51.152333    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz"
Nov 03 18:09:54.383906 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.383812    6167 generic.go:356] "Generic (PLEG): container finished" podID="66465677-7842-4438-8e69-cbfbd6334744" containerID="28512e13b9531920549f08bdfc559b8eb1fea007173d7259f4f87b19612656de" exitCode=0
Nov 03 18:09:54.384342 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.384041    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" event={"ID":"66465677-7842-4438-8e69-cbfbd6334744","Type":"ContainerDied","Data":"28512e13b9531920549f08bdfc559b8eb1fea007173d7259f4f87b19612656de"}
Nov 03 18:09:54.384342 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.384104    6167 scope.go:122] "RemoveContainer" containerID="55fca55b629e5839baaa6576d489d4eaaeaee12e5eb3fa101bf924154f0c237a"
Nov 03 18:09:54.441961 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.441929    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:09:54.506639 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.506607    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.506794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.506665    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.506848 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.506709    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.506943 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.506918    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:54.507010 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.506938    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.507066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507022    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.507121 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507085    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:54.507200 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507184    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.507264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507228    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.507400 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507382    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.507477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507417    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data\") pod \"66465677-7842-4438-8e69-cbfbd6334744\" (UID: \"66465677-7842-4438-8e69-cbfbd6334744\") "
Nov 03 18:09:54.507535 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507505    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:54.507758 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507669    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:54.507864 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.507765    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:54.508158 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.508140    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.508250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.508160    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.508250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.508169    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.508250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.508179    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66465677-7842-4438-8e69-cbfbd6334744-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.508250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.508188    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.509108 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.509080    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:54.509226 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.509153    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:54.509907 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.509886    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "kube-api-access-jcxth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:54.510336 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.510317    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy" pod "66465677-7842-4438-8e69-cbfbd6334744" (UID: "66465677-7842-4438-8e69-cbfbd6334744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:54.525146 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:09:54.525119    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:09:54.610151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.610115    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxth\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-kube-api-access-jcxth\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.610151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.610150    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66465677-7842-4438-8e69-cbfbd6334744-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.610151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.610160    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66465677-7842-4438-8e69-cbfbd6334744-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:54.610387 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:54.610170    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66465677-7842-4438-8e69-cbfbd6334744-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:09:55.392967 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:55.392935    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"
Nov 03 18:09:55.393409 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:55.392972    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc" event={"ID":"66465677-7842-4438-8e69-cbfbd6334744","Type":"ContainerDied","Data":"4ba1ba07cb451c084be1a408382981396c2d460fed48708e0941a4671518156d"}
Nov 03 18:09:55.393409 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:55.393033    6167 scope.go:122] "RemoveContainer" containerID="28512e13b9531920549f08bdfc559b8eb1fea007173d7259f4f87b19612656de"
Nov 03 18:09:55.614182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:55.614147    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"]
Nov 03 18:09:55.618567 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:55.618535    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4b6e7259-openshift-default-79f974488f-wczcc"]
Nov 03 18:09:56.930691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:09:56.930658    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66465677-7842-4438-8e69-cbfbd6334744" path="/var/lib/kubelet/pods/66465677-7842-4438-8e69-cbfbd6334744/volumes"
Nov 03 18:10:03.695071 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695037    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"]
Nov 03 18:10:03.695823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695806    6167 cpu_manager.go:414] "containerMap: removing container" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy"
Nov 03 18:10:03.695888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695824    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84743169-5dbc-428a-911c-d460225c12df" containerName="istio-proxy"
Nov 03 18:10:03.695888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695836    6167 cpu_manager.go:414] "containerMap: removing container" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy"
Nov 03 18:10:03.695888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695842    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy"
Nov 03 18:10:03.695888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695867    6167 cpu_manager.go:414] "containerMap: removing container" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy"
Nov 03 18:10:03.695888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695873    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy"
Nov 03 18:10:03.696060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.695988    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy"
Nov 03 18:10:03.696060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.696004    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="66465677-7842-4438-8e69-cbfbd6334744" containerName="istio-proxy"
Nov 03 18:10:03.699762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.699745    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.753079 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.753044    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-c91bee-serving-cert\""
Nov 03 18:10:03.755691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.755660    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"]
Nov 03 18:10:03.787119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.787083    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.787315 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.787247    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkpj\" (UniqueName: \"kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.787468 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.787440    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.787585 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.787562    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.889491 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.889451    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.889687 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.889583    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkpj\" (UniqueName: \"kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.889687 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:03.889608    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-serving-cert: secret "maas-api-e2e-shared-a-w4-c91bee-serving-cert" not found
Nov 03 18:10:03.889687 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:03.889677    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls podName:a00dea4a-0474-4f93-bfe9-d9b3046cc537 nodeName:}" failed. No retries permitted until 2026-11-03 18:10:04.389656613 +0000 UTC m=+1459.619981676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls") pod "maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" (UID: "a00dea4a-0474-4f93-bfe9-d9b3046cc537") : secret "maas-api-e2e-shared-a-w4-c91bee-serving-cert" not found
Nov 03 18:10:03.889932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.889774    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.889932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.889878    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.890778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.890691    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.893042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.893017    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:03.952397 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:03.952323    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkpj\" (UniqueName: \"kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:04.398469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:04.398431    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:04.400929 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:04.400903    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:04.615960 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:04.615926    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:05.007278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:05.007242    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"]
Nov 03 18:10:05.492070 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:05.492029    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" event={"ID":"a00dea4a-0474-4f93-bfe9-d9b3046cc537","Type":"ContainerStarted","Data":"cf3d172dbb16461045b09d1df82d9f11a042a07f19b393610f034180d8c3f370"}
Nov 03 18:10:06.501038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:06.501000    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" event={"ID":"a00dea4a-0474-4f93-bfe9-d9b3046cc537","Type":"ContainerStarted","Data":"835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef"}
Nov 03 18:10:06.501439 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:06.501082    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:06.624182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:06.624118    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" podStartSLOduration=2.798536062 podStartE2EDuration="3.624103634s" totalImagesPullingTime="825.567572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:06.619816194 +0000 UTC m=+1461.850141266" watchObservedRunningTime="2026-11-03 18:10:06.624103634 +0000 UTC m=+1461.854428705"
Nov 03 18:10:08.635656 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.635622    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"]
Nov 03 18:10:08.640418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.640399    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.697212 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.697177    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"]
Nov 03 18:10:08.744384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.744353    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5ffd208e-serving-cert\""
Nov 03 18:10:08.777510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.777462    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czclc\" (UniqueName: \"kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.778016 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.777516    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.778016 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.777604    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.778016 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.777780    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.886938    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czclc\" (UniqueName: \"kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.886990    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.887079    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.887256    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:08.887640    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-serving-cert: secret "maas-api-e2e-derive-5ffd208e-serving-cert" not found
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:08.887711    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls podName:171db62f-4b16-490b-a875-ecdac1d995c3 nodeName:}" failed. No retries permitted until 2026-11-03 18:10:09.387687801 +0000 UTC m=+1464.618012871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls") pod "maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" (UID: "171db62f-4b16-490b-a875-ecdac1d995c3") : secret "maas-api-e2e-derive-5ffd208e-serving-cert" not found
Nov 03 18:10:08.895786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.891000    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.924412 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.924341    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:08.957037 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:08.953872    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czclc\" (UniqueName: \"kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:09.396092 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:09.396056    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:09.431774 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:09.429328    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\") pod \"maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:09.577752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:09.563889    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:10.067209 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:10:10.067169    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171db62f_4b16_490b_a875_ecdac1d995c3.slice/crio-3b49a70e27ef74c29e42eabf2cded58b591f66c793f7e29116b7dc3add1cf261 WatchSource:0}: Error finding container 3b49a70e27ef74c29e42eabf2cded58b591f66c793f7e29116b7dc3add1cf261: Status 404 returned error can't find the container with id 3b49a70e27ef74c29e42eabf2cded58b591f66c793f7e29116b7dc3add1cf261
Nov 03 18:10:10.129745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:10.124524    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"]
Nov 03 18:10:10.606651 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:10.606613    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" event={"ID":"171db62f-4b16-490b-a875-ecdac1d995c3","Type":"ContainerStarted","Data":"3b49a70e27ef74c29e42eabf2cded58b591f66c793f7e29116b7dc3add1cf261"}
Nov 03 18:10:11.627230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:11.627192    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" event={"ID":"171db62f-4b16-490b-a875-ecdac1d995c3","Type":"ContainerStarted","Data":"25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e"}
Nov 03 18:10:11.627770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:11.627282    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:12.515686 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:12.515657    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:10:12.635945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:12.635893    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" podStartSLOduration=3.703672397 podStartE2EDuration="4.635878663s" totalImagesPullingTime="932.206266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:11.808024771 +0000 UTC m=+1467.038349845" watchObservedRunningTime="2026-11-03 18:10:12.635878663 +0000 UTC m=+1467.866203738"
Nov 03 18:10:16.525302 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:16.525267    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:10:17.699024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:17.698975    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:10:24.522868 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:24.522826    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:10:40.110423 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.110387    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"]
Nov 03 18:10:40.115303 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.115283    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.173666 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.173638    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"]
Nov 03 18:10:40.218414 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.218380    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b6cb8403-serving-cert\""
Nov 03 18:10:40.255435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.255380    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.255764 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.255745    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.255949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.255933    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.256090 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.256076    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mb8\" (UniqueName: \"kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.362036 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.361952    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.362036 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.362020    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.362267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.362067    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.362267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.362093    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mb8\" (UniqueName: \"kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.371472 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:40.362663    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-serving-cert: secret "maas-api-e2e-mt-b6cb8403-serving-cert" not found
Nov 03 18:10:40.375488 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:40.375464    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls podName:7fbc9105-f7fb-4d68-a414-e850011538c2 nodeName:}" failed. No retries permitted until 2026-11-03 18:10:40.875440062 +0000 UTC m=+1496.105765117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls") pod "maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" (UID: "7fbc9105-f7fb-4d68-a414-e850011538c2") : secret "maas-api-e2e-mt-b6cb8403-serving-cert" not found
Nov 03 18:10:40.375796 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.374057    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.396070 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.395897    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.425697 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.425649    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mb8\" (UniqueName: \"kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.973241 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.973206    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:40.975812 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:40.975783    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\") pod \"maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:41.030358 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:41.030325    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:41.409884 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:10:41.409846    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbc9105_f7fb_4d68_a414_e850011538c2.slice/crio-6ff0f76d2245814b194fc720c7aa32266a7730cfe481468af1243d426f877245 WatchSource:0}: Error finding container 6ff0f76d2245814b194fc720c7aa32266a7730cfe481468af1243d426f877245: Status 404 returned error can't find the container with id 6ff0f76d2245814b194fc720c7aa32266a7730cfe481468af1243d426f877245
Nov 03 18:10:41.412872 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:41.412850    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"]
Nov 03 18:10:41.974070 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:41.974033    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" event={"ID":"7fbc9105-f7fb-4d68-a414-e850011538c2","Type":"ContainerStarted","Data":"6ff0f76d2245814b194fc720c7aa32266a7730cfe481468af1243d426f877245"}
Nov 03 18:10:42.991740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:42.987499    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" event={"ID":"7fbc9105-f7fb-4d68-a414-e850011538c2","Type":"ContainerStarted","Data":"6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720"}
Nov 03 18:10:42.991740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:42.987586    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:43.113469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:43.113410    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" podStartSLOduration=2.277425031 podStartE2EDuration="3.113391647s" totalImagesPullingTime="835.966616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:43.111933682 +0000 UTC m=+1498.342258760" watchObservedRunningTime="2026-11-03 18:10:43.113391647 +0000 UTC m=+1498.343716720"
Nov 03 18:10:45.249460 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:45.249426    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:10:45.253596 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:45.253573    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:10:49.007325 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:49.007293    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:10:54.522101 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:54.522057    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:10:58.314426 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.314389    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"]
Nov 03 18:10:58.320611 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.320587    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.373524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.373482    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1720c00c-serving-cert\""
Nov 03 18:10:58.375694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.375655    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"]
Nov 03 18:10:58.387754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.387711    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.387888 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.387863    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbh7\" (UniqueName: \"kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.387965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.387890    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.388137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.388119    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.489700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.489664    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.489897 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.489812    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbh7\" (UniqueName: \"kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.489897 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:58.489820    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-serving-cert: secret "maas-api-e2e-mt-1720c00c-serving-cert" not found
Nov 03 18:10:58.489897 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.489850    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.489897 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:10:58.489892    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls podName:8a4a62d6-14de-4a05-a56d-ca5cdb832e8d nodeName:}" failed. No retries permitted until 2026-11-03 18:10:58.989870746 +0000 UTC m=+1514.220195806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls") pod "maas-api-e2e-mt-1720c00c-6977b66858-trs9w" (UID: "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d") : secret "maas-api-e2e-mt-1720c00c-serving-cert" not found
Nov 03 18:10:58.490165 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.490145    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.490853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.490832    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.492369 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.492345    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.554581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.554416    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbh7\" (UniqueName: \"kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:58.998980 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:58.998945    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:59.001838 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:59.001809    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-trs9w\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:59.235561 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:59.235528    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:10:59.612997 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:10:59.612968    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4a62d6_14de_4a05_a56d_ca5cdb832e8d.slice/crio-5619ea6d5eac622d87b1f3da9aac71a7835237996b0412e1afbdf8d14769f902 WatchSource:0}: Error finding container 5619ea6d5eac622d87b1f3da9aac71a7835237996b0412e1afbdf8d14769f902: Status 404 returned error can't find the container with id 5619ea6d5eac622d87b1f3da9aac71a7835237996b0412e1afbdf8d14769f902
Nov 03 18:10:59.617600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:10:59.617572    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"]
Nov 03 18:11:00.143187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:00.143128    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w" event={"ID":"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d","Type":"ContainerStarted","Data":"5619ea6d5eac622d87b1f3da9aac71a7835237996b0412e1afbdf8d14769f902"}
Nov 03 18:11:01.152813 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:01.152768    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w" event={"ID":"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d","Type":"ContainerStarted","Data":"d079b8453a3d3a493e1ac0931845a227e5c63d1a751a0dd766a3ad5e912cdffe"}
Nov 03 18:11:01.153219 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:01.152900    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:11:01.275277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:01.275210    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w" podStartSLOduration=2.410442554 podStartE2EDuration="3.275191425s" totalImagesPullingTime="864.748871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:01.270967681 +0000 UTC m=+1516.501292754" watchObservedRunningTime="2026-11-03 18:11:01.275191425 +0000 UTC m=+1516.505516499"
Nov 03 18:11:07.168161 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:07.168129    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:11:24.523276 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:11:24.523186    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:11:37.067748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.061341    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"]
Nov 03 18:11:37.067748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.067232    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.127777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.127741    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"]
Nov 03 18:11:37.173203 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.168987    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.173203 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.169156    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcbr\" (UniqueName: \"kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr\") pod \"payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.173203 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.172440    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7a7c76\""
Nov 03 18:11:37.173203 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.172556    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7a7c76-dockercfg-xd5h7\""
Nov 03 18:11:37.270642 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.270608    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.270857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.270771    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcbr\" (UniqueName: \"kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr\") pod \"payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.271418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.271395    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.336083 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.336005    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcbr\" (UniqueName: \"kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr\") pod \"payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:37.390827 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:37.390797    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:38.045440 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:38.045412    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"]
Nov 03 18:11:38.525151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:38.525113    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5" event={"ID":"a912bc62-c39d-4a84-8092-055d55a2f59f","Type":"ContainerStarted","Data":"dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b"}
Nov 03 18:11:38.525641 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:38.525176    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5" event={"ID":"a912bc62-c39d-4a84-8092-055d55a2f59f","Type":"ContainerStarted","Data":"6906be491f383d4e197ef80ae105c123ac443e4b18768fba9bede991bbddaf11"}
Nov 03 18:11:38.525641 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:38.525204    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:38.648337 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:38.648272    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5" podStartSLOduration=1.64825361 podStartE2EDuration="1.64825361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:38.643303466 +0000 UTC m=+1553.873628539" watchObservedRunningTime="2026-11-03 18:11:38.64825361 +0000 UTC m=+1553.878578683"
Nov 03 18:11:40.278424 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:40.278373    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:11:49.535184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:11:49.535155    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:11:54.522996 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:11:54.522958    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:12:01.173278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.173238    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"]
Nov 03 18:12:01.173813 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.173540    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" podUID="171db62f-4b16-490b-a875-ecdac1d995c3" containerName="maas-api" containerID="cri-o://25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e" gracePeriod=30
Nov 03 18:12:01.583595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.583564    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:12:01.747448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.747338    6167 generic.go:356] "Generic (PLEG): container finished" podID="171db62f-4b16-490b-a875-ecdac1d995c3" containerID="25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e" exitCode=0
Nov 03 18:12:01.747816 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.747790    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" event={"ID":"171db62f-4b16-490b-a875-ecdac1d995c3","Type":"ContainerDied","Data":"25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e"}
Nov 03 18:12:01.748041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.748022    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb" event={"ID":"171db62f-4b16-490b-a875-ecdac1d995c3","Type":"ContainerDied","Data":"3b49a70e27ef74c29e42eabf2cded58b591f66c793f7e29116b7dc3add1cf261"}
Nov 03 18:12:01.748187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.748174    6167 scope.go:122] "RemoveContainer" containerID="25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e"
Nov 03 18:12:01.748675 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.748653    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"
Nov 03 18:12:01.787519 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.787474    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc\" (UniqueName: \"kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc\") pod \"171db62f-4b16-490b-a875-ecdac1d995c3\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") "
Nov 03 18:12:01.787760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.787567    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\") pod \"171db62f-4b16-490b-a875-ecdac1d995c3\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") "
Nov 03 18:12:01.787854 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.787819    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs\") pod \"171db62f-4b16-490b-a875-ecdac1d995c3\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") "
Nov 03 18:12:01.788049 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.788018    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca\") pod \"171db62f-4b16-490b-a875-ecdac1d995c3\" (UID: \"171db62f-4b16-490b-a875-ecdac1d995c3\") "
Nov 03 18:12:01.788635 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.788605    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca" pod "171db62f-4b16-490b-a875-ecdac1d995c3" (UID: "171db62f-4b16-490b-a875-ecdac1d995c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:01.789407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.789366    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/171db62f-4b16-490b-a875-ecdac1d995c3-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:01.795998 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.795969    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls" pod "171db62f-4b16-490b-a875-ecdac1d995c3" (UID: "171db62f-4b16-490b-a875-ecdac1d995c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:01.796135 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.796121    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs" pod "171db62f-4b16-490b-a875-ecdac1d995c3" (UID: "171db62f-4b16-490b-a875-ecdac1d995c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:01.797784 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.797704    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc" pod "171db62f-4b16-490b-a875-ecdac1d995c3" (UID: "171db62f-4b16-490b-a875-ecdac1d995c3"). InnerVolumeSpecName "kube-api-access-czclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:01.804221 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.803858    6167 scope.go:122] "RemoveContainer" containerID="25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e"
Nov 03 18:12:01.804547 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:01.804453    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e\": container with ID starting with 25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e not found: ID does not exist" containerID="25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e"
Nov 03 18:12:01.804547 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.804491    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e"} err="failed to get container status \"25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e\": rpc error: code = NotFound desc = could not find container \"25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e\": container with ID starting with 25b648dbeff2140a5a707521c5a8ff649930108c3b2711155562c1e29760ed5e not found: ID does not exist"
Nov 03 18:12:01.905740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.892774    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czclc\" (UniqueName: \"kubernetes.io/projected/171db62f-4b16-490b-a875-ecdac1d995c3-kube-api-access-czclc\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:01.905740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.892815    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:01.905740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:01.892831    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/171db62f-4b16-490b-a875-ecdac1d995c3-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:02.181162 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.181124    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"]
Nov 03 18:12:02.229768 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.229696    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ffd208e-7bbc5b74b8-vcffb"]
Nov 03 18:12:02.695417 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.695382    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"]
Nov 03 18:12:02.696121 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.696107    6167 cpu_manager.go:414] "containerMap: removing container" podUID="171db62f-4b16-490b-a875-ecdac1d995c3" containerName="maas-api"
Nov 03 18:12:02.696190 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.696122    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171db62f-4b16-490b-a875-ecdac1d995c3" containerName="maas-api"
Nov 03 18:12:02.696282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.696272    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="171db62f-4b16-490b-a875-ecdac1d995c3" containerName="maas-api"
Nov 03 18:12:02.699838 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.699820    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.753244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.753210    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-260dd0-serving-cert\""
Nov 03 18:12:02.759053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.759021    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"]
Nov 03 18:12:02.814242 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.814209    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.814425 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.814357    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf699\" (UniqueName: \"kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.814565 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.814543    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.814602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.814580    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.916304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.916271    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.916472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.916447    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.916534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.916522    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf699\" (UniqueName: \"kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.916612 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:02.916586    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-serving-cert: secret "maas-api-e2e-shared-b-w4-260dd0-serving-cert" not found
Nov 03 18:12:02.916769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.916634    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.916769 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:02.916648    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls podName:7ca50e99-73fa-4476-9fe3-abcdcc4b92a1 nodeName:}" failed. No retries permitted until 2026-11-03 18:12:03.41663276 +0000 UTC m=+1578.646957813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls") pod "maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" (UID: "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1") : secret "maas-api-e2e-shared-b-w4-260dd0-serving-cert" not found
Nov 03 18:12:02.917098 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.917073    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.919114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.919095    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:02.930345 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.930306    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171db62f-4b16-490b-a875-ecdac1d995c3" path="/var/lib/kubelet/pods/171db62f-4b16-490b-a875-ecdac1d995c3/volumes"
Nov 03 18:12:02.978667 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:02.978595    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf699\" (UniqueName: \"kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:03.430533 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:03.430491    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:03.433043 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:03.433012    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:03.615262 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:03.615215    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:04.015833 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:12:04.015798    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca50e99_73fa_4476_9fe3_abcdcc4b92a1.slice/crio-f2e2917aed7978da0ad22634275655075e2d3e51b7e80ccccfc87e9d9d3b711f WatchSource:0}: Error finding container f2e2917aed7978da0ad22634275655075e2d3e51b7e80ccccfc87e9d9d3b711f: Status 404 returned error can't find the container with id f2e2917aed7978da0ad22634275655075e2d3e51b7e80ccccfc87e9d9d3b711f
Nov 03 18:12:04.017915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:04.017889    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:12:04.020484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:04.019896    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"]
Nov 03 18:12:04.778782 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:04.778749    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" event={"ID":"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1","Type":"ContainerStarted","Data":"f2e2917aed7978da0ad22634275655075e2d3e51b7e80ccccfc87e9d9d3b711f"}
Nov 03 18:12:05.789497 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:05.789462    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" event={"ID":"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1","Type":"ContainerStarted","Data":"8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8"}
Nov 03 18:12:05.790006 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:05.789564    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:05.912067 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:05.912013    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" podStartSLOduration=3.03150778 podStartE2EDuration="3.911999235s" totalImagesPullingTime="880.491455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:05.908401513 +0000 UTC m=+1581.138726585" watchObservedRunningTime="2026-11-03 18:12:05.911999235 +0000 UTC m=+1581.142324307"
Nov 03 18:12:11.804341 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:11.804311    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:12:14.452405 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.452368    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"]
Nov 03 18:12:14.452898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.452627    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy" containerID="cri-o://0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82" gracePeriod=30
Nov 03 18:12:14.502706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.502669    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"]
Nov 03 18:12:14.503011 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.502968    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" containerID="cri-o://e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d" gracePeriod=30
Nov 03 18:12:14.508461 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.508430    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"]
Nov 03 18:12:14.508703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.508675    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" containerID="cri-o://9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0" gracePeriod=30
Nov 03 18:12:14.940278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.940239    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"]
Nov 03 18:12:14.940470 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.940438    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy" containerID="cri-o://b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f" gracePeriod=30
Nov 03 18:12:14.945479 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.945443    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"]
Nov 03 18:12:14.945711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.945664    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy" containerID="cri-o://0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c" gracePeriod=30
Nov 03 18:12:14.950131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.950107    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"]
Nov 03 18:12:14.950309 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:14.950263    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy" containerID="cri-o://12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab" gracePeriod=30
Nov 03 18:12:18.279712 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.279675    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"]
Nov 03 18:12:18.286737 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.286700    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.342431 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.342395    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"]
Nov 03 18:12:18.385622 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.385584    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-80aca1-dockercfg-m7wl8\""
Nov 03 18:12:18.391668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.386318    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-80aca1\""
Nov 03 18:12:18.431504 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.431460    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"]
Nov 03 18:12:18.433266 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.433225    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9zs\" (UniqueName: \"kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs\") pod \"payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.433453 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.433427    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.437541 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.437521    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.489146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.489108    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"]
Nov 03 18:12:18.535354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.535254    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.535537 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.535402    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9zs\" (UniqueName: \"kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs\") pod \"payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.535712 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.535556    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.535712 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.535668    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zcq\" (UniqueName: \"kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq\") pod \"payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.537185 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.537157    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.599430 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.599394    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9zs\" (UniqueName: \"kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs\") pod \"payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.604564 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.604531    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:18.605035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.604945    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:12:18.605035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.605010    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:12:18.639196 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.639152    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zcq\" (UniqueName: \"kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq\") pod \"payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.641024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.641003    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.642123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.641972    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.712738 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.712529    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zcq\" (UniqueName: \"kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq\") pod \"payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.802788 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.802078    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:18.967190 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.967151    6167 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:12:18.967378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:18.967232    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:12:19.154456 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.154428    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"]
Nov 03 18:12:19.339759 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:12:19.339713    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de76cfd_889f_4565_9f60_3434798f70d7.slice/crio-d6496d2e6b0b3a28dc82b9aa83dfa9c5ab165c7c1fcf78a396b4a543a8d887ff WatchSource:0}: Error finding container d6496d2e6b0b3a28dc82b9aa83dfa9c5ab165c7c1fcf78a396b4a543a8d887ff: Status 404 returned error can't find the container with id d6496d2e6b0b3a28dc82b9aa83dfa9c5ab165c7c1fcf78a396b4a543a8d887ff
Nov 03 18:12:19.341958 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.341934    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"]
Nov 03 18:12:19.874789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.874703    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:12:19.885929 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.885898    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:12:19.910757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.910732    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:12:19.922362 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.922334    6167 generic.go:356] "Generic (PLEG): container finished" podID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerID="0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82" exitCode=0
Nov 03 18:12:19.922487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.922369    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"
Nov 03 18:12:19.922487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.922411    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" event={"ID":"acaf5374-f308-4ee6-aa7c-05cebdc4d271","Type":"ContainerDied","Data":"0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82"}
Nov 03 18:12:19.922487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.922459    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg" event={"ID":"acaf5374-f308-4ee6-aa7c-05cebdc4d271","Type":"ContainerDied","Data":"e7946c7592fa0db9aa9f3847d9a45bd3f5287ea306763c01f1e0efdbd5a963f2"}
Nov 03 18:12:19.922487 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.922486    6167 scope.go:122] "RemoveContainer" containerID="0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82"
Nov 03 18:12:19.923690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923660    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.923802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923777    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.923871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923814    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.923871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923846    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.923973 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923886    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.923973 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923909    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.924069 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.923998    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.924069 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924025    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.924173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924072    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.924173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924079    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:19.924173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924123    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.924173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924154    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.924354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924188    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.924354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924228    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.924354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924261    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.924354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924287    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.924354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924316    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.924577 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924372    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.924577 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924399    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.924577 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924440    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw" event={"ID":"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7","Type":"ContainerStarted","Data":"6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf"}
Nov 03 18:12:19.924753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924612    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw" event={"ID":"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7","Type":"ContainerStarted","Data":"37753eda0452ca6f1dbe7add225cbdae0c62315c5ce44e2926eb5fb0a1d20832"}
Nov 03 18:12:19.924753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924530    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925172    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.924459    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925357    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925368    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925414    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925455    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925488    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925520    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925554    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket\") pod \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\" (UID: \"9c3a90e0-1e22-40c7-8b52-d732ee0c6903\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925587    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925615    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy\") pod \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\" (UID: \"acaf5374-f308-4ee6-aa7c-05cebdc4d271\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925700    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925714    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925849    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.926114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.925896    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert\") pod \"66b89db3-b00c-49cd-af19-08b5d49b6712\" (UID: \"66b89db3-b00c-49cd-af19-08b5d49b6712\") "
Nov 03 18:12:19.926924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.926127    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:19.926924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.926367    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.926549    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.926924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.926556    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.927128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.926936    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.927128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.926960    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.927128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927008    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:19.927397 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927380    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.927484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927408    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.927484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927425    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.927484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927441    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.927484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927455    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.927484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927468    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927496    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927508    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927521    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927533    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927544    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927462    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "kube-api-access-xr978". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927930    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.927939    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.928205    6167 generic.go:356] "Generic (PLEG): container finished" podID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerID="e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d" exitCode=0
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.928232    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.928275    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" event={"ID":"9c3a90e0-1e22-40c7-8b52-d732ee0c6903","Type":"ContainerDied","Data":"e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d"}
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.928322    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl" event={"ID":"9c3a90e0-1e22-40c7-8b52-d732ee0c6903","Type":"ContainerDied","Data":"bf1347f31220e0d9faa82cb914d071220606e6dbfc80ba132c68e98d4d790309"}
Nov 03 18:12:19.929146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.928965    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:19.930994 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.930958    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.931098 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.930995    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "kube-api-access-mr589". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:19.931098 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.931028    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "kube-api-access-7txbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:19.931229 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.931143    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.931465 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.931445    6167 generic.go:356] "Generic (PLEG): container finished" podID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerID="9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0" exitCode=0
Nov 03 18:12:19.931535 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.931479    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"
Nov 03 18:12:19.931589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.931545    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" event={"ID":"66b89db3-b00c-49cd-af19-08b5d49b6712","Type":"ContainerDied","Data":"9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0"}
Nov 03 18:12:19.931589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.931583    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b" event={"ID":"66b89db3-b00c-49cd-af19-08b5d49b6712","Type":"ContainerDied","Data":"ae2fc21196375870080e3bcfcb8d252abf8fd07ed0922d7a1836fd7a925c750d"}
Nov 03 18:12:19.932356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.932333    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:19.933258 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.933233    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm" event={"ID":"4de76cfd-889f-4565-9f60-3434798f70d7","Type":"ContainerStarted","Data":"85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270"}
Nov 03 18:12:19.933375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.933278    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm" event={"ID":"4de76cfd-889f-4565-9f60-3434798f70d7","Type":"ContainerStarted","Data":"d6496d2e6b0b3a28dc82b9aa83dfa9c5ab165c7c1fcf78a396b4a543a8d887ff"}
Nov 03 18:12:19.933433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.933398    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:19.937000 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.936803    6167 scope.go:122] "RemoveContainer" containerID="0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82"
Nov 03 18:12:19.937000 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.936954    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:19.937000 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.936973    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo" pod "66b89db3-b00c-49cd-af19-08b5d49b6712" (UID: "66b89db3-b00c-49cd-af19-08b5d49b6712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:19.937000 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.936993    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token" pod "9c3a90e0-1e22-40c7-8b52-d732ee0c6903" (UID: "9c3a90e0-1e22-40c7-8b52-d732ee0c6903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:19.937231 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:19.937071    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82\": container with ID starting with 0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82 not found: ID does not exist" containerID="0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82"
Nov 03 18:12:19.937231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.937091    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82"} err="failed to get container status \"0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82\": rpc error: code = NotFound desc = could not find container \"0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82\": container with ID starting with 0f44ee9d5a6b7873d390c8caf2d642eab12a8d8e287a7ef0dcf3bca23d0a7d82 not found: ID does not exist"
Nov 03 18:12:19.937231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.937114    6167 scope.go:122] "RemoveContainer" containerID="e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d"
Nov 03 18:12:19.938782 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.938703    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token" pod "acaf5374-f308-4ee6-aa7c-05cebdc4d271" (UID: "acaf5374-f308-4ee6-aa7c-05cebdc4d271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:19.950100 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.950077    6167 scope.go:122] "RemoveContainer" containerID="e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d"
Nov 03 18:12:19.950423 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:19.950397    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d\": container with ID starting with e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d not found: ID does not exist" containerID="e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d"
Nov 03 18:12:19.950496 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.950434    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d"} err="failed to get container status \"e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d\": rpc error: code = NotFound desc = could not find container \"e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d\": container with ID starting with e578f5b5e34bc78117205e034de2e28ff6116c289d7889f48e6b5717beca2c5d not found: ID does not exist"
Nov 03 18:12:19.950496 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.950459    6167 scope.go:122] "RemoveContainer" containerID="9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0"
Nov 03 18:12:19.964893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.964869    6167 scope.go:122] "RemoveContainer" containerID="9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0"
Nov 03 18:12:19.965226 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:19.965205    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0\": container with ID starting with 9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0 not found: ID does not exist" containerID="9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0"
Nov 03 18:12:19.965290 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:19.965237    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0"} err="failed to get container status \"9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0\": rpc error: code = NotFound desc = could not find container \"9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0\": container with ID starting with 9d3178ef811d1d9145cbd44927e4ee53e92ed65bad564d213dcdabd57acdefa0 not found: ID does not exist"
Nov 03 18:12:20.028870 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028845    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.028870 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028871    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr589\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-kube-api-access-mr589\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.028870 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028881    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txbn\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-kube-api-access-7txbn\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028890    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028898    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028910    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028920    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028929    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028937    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf5374-f308-4ee6-aa7c-05cebdc4d271-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028946    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028954    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b89db3-b00c-49cd-af19-08b5d49b6712-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028963    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b89db3-b00c-49cd-af19-08b5d49b6712-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028971    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028980    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr978\" (UniqueName: \"kubernetes.io/projected/66b89db3-b00c-49cd-af19-08b5d49b6712-kube-api-access-xr978\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028988    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.029057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.028997    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3a90e0-1e22-40c7-8b52-d732ee0c6903-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.295356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.295306    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw" podStartSLOduration=2.295291104 podStartE2EDuration="2.295291104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:20.293221433 +0000 UTC m=+1595.523546502" watchObservedRunningTime="2026-11-03 18:12:20.295291104 +0000 UTC m=+1595.525616206"
Nov 03 18:12:20.303132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.303107    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:12:20.335878 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.335839    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.336052 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.335964    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.336052 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.336022    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.336161 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.336078    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.336457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.336429    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.337173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.337147    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.337284 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.337223    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.337339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.337330    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.337397 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.337381    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.337446 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.337397    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:20.337500 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.337458    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data\") pod \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\" (UID: \"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c\") "
Nov 03 18:12:20.338385 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.338347    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.338534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.338504    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.338857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.338839    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.339056 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.339013    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.339159 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.339076    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.339236 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.339222    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.341685 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.341659    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:20.342039 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.341733    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "kube-api-access-7gxvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:20.342567 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.342541    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token" pod "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" (UID: "ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440667    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440695    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440705    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440728    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440740    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxvk\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-kube-api-access-7gxvk\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440749    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.442534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.440758    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.462426 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.462393    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:12:20.467975 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.467937    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:12:20.541691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.541659    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.541939 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.541920    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.541980    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542095 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542037    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542095 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542071    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542117    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542148    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542252    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542344    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542372    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542457    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542490    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542548    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542697 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542589    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542697 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542681    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542829 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542762    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo\") pod \"a09609ee-8a67-446d-90dc-6a841437ae6b\" (UID: \"a09609ee-8a67-446d-90dc-6a841437ae6b\") "
Nov 03 18:12:20.542883 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542828    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.542932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.542901    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket\") pod \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\" (UID: \"9edfd3b8-2ccb-444f-913c-91bc502bc9a0\") "
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.543080    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.543311    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.543521    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.543684    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.543834    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.543870    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544249    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544683    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.544809 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544776    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.544809 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544791    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.544918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544807    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.544918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544834    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.544918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.544845    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.545074 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.545006    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.545074 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.545045    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.545525 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.545503    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:20.547167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.547121    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:20.547167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.547141    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:20.547577 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.547560    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:20.547871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.547846    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:20.548243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.548225    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:20.548589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.548570    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j" pod "9edfd3b8-2ccb-444f-913c-91bc502bc9a0" (UID: "9edfd3b8-2ccb-444f-913c-91bc502bc9a0"). InnerVolumeSpecName "kube-api-access-bsf6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:20.548864 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.548826    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv" pod "a09609ee-8a67-446d-90dc-6a841437ae6b" (UID: "a09609ee-8a67-446d-90dc-6a841437ae6b"). InnerVolumeSpecName "kube-api-access-bxntv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:20.587488 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.587413    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm" podStartSLOduration=2.587393623 podStartE2EDuration="2.587393623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:20.586886796 +0000 UTC m=+1595.817211867" watchObservedRunningTime="2026-11-03 18:12:20.587393623 +0000 UTC m=+1595.817718695"
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646020    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646042    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646052    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsf6j\" (UniqueName: \"kubernetes.io/projected/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-kube-api-access-bsf6j\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646063    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxntv\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-kube-api-access-bxntv\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646074    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646083    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646093    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646102    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edfd3b8-2ccb-444f-913c-91bc502bc9a0-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646110    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646118    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09609ee-8a67-446d-90dc-6a841437ae6b-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646126    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09609ee-8a67-446d-90dc-6a841437ae6b-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646134    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.646154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.646141    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09609ee-8a67-446d-90dc-6a841437ae6b-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:12:20.713452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.705138    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"]
Nov 03 18:12:20.718418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.718370    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-wqpgl"]
Nov 03 18:12:20.929705 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.929653    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" path="/var/lib/kubelet/pods/9c3a90e0-1e22-40c7-8b52-d732ee0c6903/volumes"
Nov 03 18:12:20.949843 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.949811    6167 generic.go:356] "Generic (PLEG): container finished" podID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerID="0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c" exitCode=0
Nov 03 18:12:20.950080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.949880    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" event={"ID":"a09609ee-8a67-446d-90dc-6a841437ae6b","Type":"ContainerDied","Data":"0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c"}
Nov 03 18:12:20.950274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.950241    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z" event={"ID":"a09609ee-8a67-446d-90dc-6a841437ae6b","Type":"ContainerDied","Data":"8f6b135e232b872342b6d103dfee0c38ccfff7c0ced0a2c581aafb87aa8e60c5"}
Nov 03 18:12:20.950274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.949939    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"
Nov 03 18:12:20.950420 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.950286    6167 scope.go:122] "RemoveContainer" containerID="0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c"
Nov 03 18:12:20.964567 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.964506    6167 generic.go:356] "Generic (PLEG): container finished" podID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerID="b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f" exitCode=0
Nov 03 18:12:20.964820 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.964797    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" event={"ID":"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c","Type":"ContainerDied","Data":"b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f"}
Nov 03 18:12:20.964979 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.964961    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh" event={"ID":"ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c","Type":"ContainerDied","Data":"87e0858e0622ac75b6e549fa2717fd3da7d2c5d9c04988aa11923280068f308e"}
Nov 03 18:12:20.967102 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.967082    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"
Nov 03 18:12:20.976522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.976502    6167 scope.go:122] "RemoveContainer" containerID="0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c"
Nov 03 18:12:20.976899 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:20.976871    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c\": container with ID starting with 0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c not found: ID does not exist" containerID="0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c"
Nov 03 18:12:20.977062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.977034    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c"} err="failed to get container status \"0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c\": rpc error: code = NotFound desc = could not find container \"0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c\": container with ID starting with 0ac85d11f08b972878ad6609d6c56eb29da184febbb404b215daa2cae6b5c06c not found: ID does not exist"
Nov 03 18:12:20.977181 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.977167    6167 scope.go:122] "RemoveContainer" containerID="b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f"
Nov 03 18:12:20.977326 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.977119    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" event={"ID":"9edfd3b8-2ccb-444f-913c-91bc502bc9a0","Type":"ContainerDied","Data":"12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab"}
Nov 03 18:12:20.977447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.977215    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"
Nov 03 18:12:20.977544 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.977100    6167 generic.go:356] "Generic (PLEG): container finished" podID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerID="12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab" exitCode=0
Nov 03 18:12:20.978099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.977707    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf" event={"ID":"9edfd3b8-2ccb-444f-913c-91bc502bc9a0","Type":"ContainerDied","Data":"ddd00a21b4f15122ff077bd2ba4cd896b17bad4ed1aae884d5badf58f92cb8d0"}
Nov 03 18:12:20.997982 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.997883    6167 scope.go:122] "RemoveContainer" containerID="b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f"
Nov 03 18:12:20.998404 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:20.998277    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f\": container with ID starting with b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f not found: ID does not exist" containerID="b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f"
Nov 03 18:12:20.998404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.998311    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f"} err="failed to get container status \"b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f\": rpc error: code = NotFound desc = could not find container \"b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f\": container with ID starting with b7734cbcecf94d74d015bbe70f4f103447238b917b5581cbde2b5cd0b87dfb8f not found: ID does not exist"
Nov 03 18:12:20.998404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:20.998334    6167 scope.go:122] "RemoveContainer" containerID="12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab"
Nov 03 18:12:21.013944 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.013905    6167 scope.go:122] "RemoveContainer" containerID="12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab"
Nov 03 18:12:21.014296 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:21.014266    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab\": container with ID starting with 12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab not found: ID does not exist" containerID="12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab"
Nov 03 18:12:21.014386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.014306    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab"} err="failed to get container status \"12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab\": rpc error: code = NotFound desc = could not find container \"12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab\": container with ID starting with 12f07c7bf75ff3f037cb7546894a3d8f5fc0fa867ede484a6a68357f1615a1ab not found: ID does not exist"
Nov 03 18:12:21.088749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.087754    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"]
Nov 03 18:12:21.091779 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.091705    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-rfs6b"]
Nov 03 18:12:21.322876 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.322847    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"]
Nov 03 18:12:21.327883 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.327851    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-dd5hg"]
Nov 03 18:12:21.436769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.436694    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"]
Nov 03 18:12:21.440388 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.440360    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-bntrf"]
Nov 03 18:12:21.551379 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.551344    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"]
Nov 03 18:12:21.558398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.558369    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-sqg2z"]
Nov 03 18:12:21.667425 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.667339    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"]
Nov 03 18:12:21.671024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:21.670991    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-svdnh"]
Nov 03 18:12:22.929992 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:22.929962    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" path="/var/lib/kubelet/pods/66b89db3-b00c-49cd-af19-08b5d49b6712/volumes"
Nov 03 18:12:22.930464 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:22.930446    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" path="/var/lib/kubelet/pods/9edfd3b8-2ccb-444f-913c-91bc502bc9a0/volumes"
Nov 03 18:12:22.930929 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:22.930916    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" path="/var/lib/kubelet/pods/a09609ee-8a67-446d-90dc-6a841437ae6b/volumes"
Nov 03 18:12:22.931670 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:22.931656    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" path="/var/lib/kubelet/pods/acaf5374-f308-4ee6-aa7c-05cebdc4d271/volumes"
Nov 03 18:12:22.932131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:22.932118    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" path="/var/lib/kubelet/pods/ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c/volumes"
Nov 03 18:12:24.521940 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:24.521904    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:12:30.579389 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.579356    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"]
Nov 03 18:12:30.580139 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580121    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy"
Nov 03 18:12:30.580139 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580139    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580154    6167 cpu_manager.go:414] "containerMap: removing container" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580160    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580186    6167 cpu_manager.go:414] "containerMap: removing container" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580191    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580198    6167 cpu_manager.go:414] "containerMap: removing container" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580203    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580214    6167 cpu_manager.go:414] "containerMap: removing container" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580219    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580230    6167 cpu_manager.go:414] "containerMap: removing container" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy"
Nov 03 18:12:30.580306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580235    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy"
Nov 03 18:12:30.580807 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580346    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6ad7a3-7eaf-4a0e-adad-70d5e4ac0e9c" containerName="istio-proxy"
Nov 03 18:12:30.580807 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580353    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09609ee-8a67-446d-90dc-6a841437ae6b" containerName="istio-proxy"
Nov 03 18:12:30.580807 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580363    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="acaf5374-f308-4ee6-aa7c-05cebdc4d271" containerName="istio-proxy"
Nov 03 18:12:30.580807 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580368    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3a90e0-1e22-40c7-8b52-d732ee0c6903" containerName="istio-proxy"
Nov 03 18:12:30.580807 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580375    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edfd3b8-2ccb-444f-913c-91bc502bc9a0" containerName="istio-proxy"
Nov 03 18:12:30.580807 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.580380    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b89db3-b00c-49cd-af19-08b5d49b6712" containerName="istio-proxy"
Nov 03 18:12:30.586012 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.585991    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.638839 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.638813    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7a7c76-serving-cert\""
Nov 03 18:12:30.703759 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.694277    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"]
Nov 03 18:12:30.710576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.710544    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.710829 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.710801    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbm49\" (UniqueName: \"kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.711031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.711008    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.711157 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.711140    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.812842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.812810    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.812842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.812852    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.813068 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.813022    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbm49\" (UniqueName: \"kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.813068 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:30.813045    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-serving-cert: secret "maas-api-e2e-shared-a-w2-7a7c76-serving-cert" not found
Nov 03 18:12:30.813202 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:30.813141    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls podName:a1a6f824-e9f5-4f47-a8f2-74b4e266c174 nodeName:}" failed. No retries permitted until 2026-11-03 18:12:31.313119321 +0000 UTC m=+1606.543444383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls") pod "maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" (UID: "a1a6f824-e9f5-4f47-a8f2-74b4e266c174") : secret "maas-api-e2e-shared-a-w2-7a7c76-serving-cert" not found
Nov 03 18:12:30.813270 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.813204    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.813608 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.813567    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.815818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.815796    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.877590 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.877509    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbm49\" (UniqueName: \"kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:30.998312 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:30.997034    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:12:31.370759 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:31.362553    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:31.420987 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:31.420955    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:31.503428 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:31.503010    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:32.022699 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:32.022627    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:12:32.323588 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:12:32.323234    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6f824_e9f5_4f47_a8f2_74b4e266c174.slice/crio-34c37157f3362fb1a03498eb7524951acb020c07721e2fdc035f123305f67a0a WatchSource:0}: Error finding container 34c37157f3362fb1a03498eb7524951acb020c07721e2fdc035f123305f67a0a: Status 404 returned error can't find the container with id 34c37157f3362fb1a03498eb7524951acb020c07721e2fdc035f123305f67a0a
Nov 03 18:12:32.426475 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:32.426351    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"]
Nov 03 18:12:33.201641 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:33.201604    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" event={"ID":"a1a6f824-e9f5-4f47-a8f2-74b4e266c174","Type":"ContainerStarted","Data":"34c37157f3362fb1a03498eb7524951acb020c07721e2fdc035f123305f67a0a"}
Nov 03 18:12:34.213501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:34.213460    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" event={"ID":"a1a6f824-e9f5-4f47-a8f2-74b4e266c174","Type":"ContainerStarted","Data":"ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f"}
Nov 03 18:12:34.214059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:34.213531    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:34.333325 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:34.333278    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" podStartSLOduration=3.45163093 podStartE2EDuration="4.333263938s" totalImagesPullingTime="881.633008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:34.332802045 +0000 UTC m=+1609.563127120" watchObservedRunningTime="2026-11-03 18:12:34.333263938 +0000 UTC m=+1609.563589063"
Nov 03 18:12:40.231408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:12:40.231378    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:12:54.523378 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:12:54.523283    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:13:07.325424 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.325388    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"]
Nov 03 18:13:07.330371 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.330344    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.386893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.386858    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"]
Nov 03 18:13:07.431290 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.431250    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.431488 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.431441    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.431648 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.431626    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx2n\" (UniqueName: \"kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.431770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.431681    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.432506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.432478    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-80aca1-serving-cert\""
Nov 03 18:13:07.533176 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.533140    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx2n\" (UniqueName: \"kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.533370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.533211    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.533370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.533274    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.533370 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:07.533347    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-serving-cert: secret "maas-api-e2e-shared-a-w5-80aca1-serving-cert" not found
Nov 03 18:13:07.533510 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:07.533423    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls podName:596176e2-4358-4834-864e-d12afa8f0282 nodeName:}" failed. No retries permitted until 2026-11-03 18:13:08.03340638 +0000 UTC m=+1643.263731433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls") pod "maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" (UID: "596176e2-4358-4834-864e-d12afa8f0282") : secret "maas-api-e2e-shared-a-w5-80aca1-serving-cert" not found
Nov 03 18:13:07.533510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.533465    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.534141 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.534114    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.536076 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.536054    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:07.679570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:07.679489    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx2n\" (UniqueName: \"kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:08.052399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:08.052309    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:08.054950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:08.054926    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:08.247713 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:08.247681    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:08.626303 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:13:08.626264    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596176e2_4358_4834_864e_d12afa8f0282.slice/crio-50de294fad6328c2d408723df7826c5e728a219a70331de34d52e967bbdb0c10 WatchSource:0}: Error finding container 50de294fad6328c2d408723df7826c5e728a219a70331de34d52e967bbdb0c10: Status 404 returned error can't find the container with id 50de294fad6328c2d408723df7826c5e728a219a70331de34d52e967bbdb0c10
Nov 03 18:13:08.628469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:08.628446    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"]
Nov 03 18:13:09.492207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:09.492172    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" event={"ID":"596176e2-4358-4834-864e-d12afa8f0282","Type":"ContainerStarted","Data":"50de294fad6328c2d408723df7826c5e728a219a70331de34d52e967bbdb0c10"}
Nov 03 18:13:10.502956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:10.502908    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" event={"ID":"596176e2-4358-4834-864e-d12afa8f0282","Type":"ContainerStarted","Data":"54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf"}
Nov 03 18:13:10.503364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:10.503160    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:10.623860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:10.623807    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" podStartSLOduration=2.665427885 podStartE2EDuration="3.623792672s" totalImagesPullingTime="958.364787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:10.622116139 +0000 UTC m=+1645.852441212" watchObservedRunningTime="2026-11-03 18:13:10.623792672 +0000 UTC m=+1645.854117743"
Nov 03 18:13:16.517226 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:16.517196    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:13:24.523558 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:24.523520    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:13:25.021701 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:25.021672    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:13:33.865609 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:33.865563    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"]
Nov 03 18:13:45.080411 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.080373    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"]
Nov 03 18:13:45.083830 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.080614    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" podUID="7fbc9105-f7fb-4d68-a414-e850011538c2" containerName="maas-api" containerID="cri-o://6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720" gracePeriod=30
Nov 03 18:13:45.331752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.331646    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"]
Nov 03 18:13:45.332889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.332855    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" containerID="cri-o://46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6" gracePeriod=30
Nov 03 18:13:45.352732 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.352690    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"]
Nov 03 18:13:45.353517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.353453    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy" containerID="cri-o://1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5" gracePeriod=30
Nov 03 18:13:45.394486 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.389780    6167 scope.go:122] "RemoveContainer" containerID="9eebce5a833f4e197b7d9da485c13809141d8e5b9ed4bf683c3f3860698ff9dc"
Nov 03 18:13:45.845714 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.845683    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:13:45.882137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.882059    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca\") pod \"7fbc9105-f7fb-4d68-a414-e850011538c2\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") "
Nov 03 18:13:45.882441 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.882408    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8\" (UniqueName: \"kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8\") pod \"7fbc9105-f7fb-4d68-a414-e850011538c2\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") "
Nov 03 18:13:45.882550 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.882462    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca" pod "7fbc9105-f7fb-4d68-a414-e850011538c2" (UID: "7fbc9105-f7fb-4d68-a414-e850011538c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:45.882611 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.882567    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs\") pod \"7fbc9105-f7fb-4d68-a414-e850011538c2\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") "
Nov 03 18:13:45.882679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.882658    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\") pod \"7fbc9105-f7fb-4d68-a414-e850011538c2\" (UID: \"7fbc9105-f7fb-4d68-a414-e850011538c2\") "
Nov 03 18:13:45.883959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.883354    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc9105-f7fb-4d68-a414-e850011538c2-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:45.889260 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.889237    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs" pod "7fbc9105-f7fb-4d68-a414-e850011538c2" (UID: "7fbc9105-f7fb-4d68-a414-e850011538c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:45.889712 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.889686    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls" pod "7fbc9105-f7fb-4d68-a414-e850011538c2" (UID: "7fbc9105-f7fb-4d68-a414-e850011538c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:45.889887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.889822    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8" pod "7fbc9105-f7fb-4d68-a414-e850011538c2" (UID: "7fbc9105-f7fb-4d68-a414-e850011538c2"). InnerVolumeSpecName "kube-api-access-v9mb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:45.907258 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.907225    6167 generic.go:356] "Generic (PLEG): container finished" podID="7fbc9105-f7fb-4d68-a414-e850011538c2" containerID="6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720" exitCode=0
Nov 03 18:13:45.907390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.907270    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"
Nov 03 18:13:45.907390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.907279    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" event={"ID":"7fbc9105-f7fb-4d68-a414-e850011538c2","Type":"ContainerDied","Data":"6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720"}
Nov 03 18:13:45.907390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.907333    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9" event={"ID":"7fbc9105-f7fb-4d68-a414-e850011538c2","Type":"ContainerDied","Data":"6ff0f76d2245814b194fc720c7aa32266a7730cfe481468af1243d426f877245"}
Nov 03 18:13:45.907390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.907364    6167 scope.go:122] "RemoveContainer" containerID="6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720"
Nov 03 18:13:45.922432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.922351    6167 scope.go:122] "RemoveContainer" containerID="6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720"
Nov 03 18:13:45.922686 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:45.922665    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720\": container with ID starting with 6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720 not found: ID does not exist" containerID="6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720"
Nov 03 18:13:45.922780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.922699    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720"} err="failed to get container status \"6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720\": rpc error: code = NotFound desc = could not find container \"6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720\": container with ID starting with 6decc0e922839f38fbed21a11e155b87d6dae4c98e7280a2f66679ab0ce4f720 not found: ID does not exist"
Nov 03 18:13:45.985621 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.985578    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:45.985621 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.985619    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbc9105-f7fb-4d68-a414-e850011538c2-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:45.985906 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:45.985635    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9mb8\" (UniqueName: \"kubernetes.io/projected/7fbc9105-f7fb-4d68-a414-e850011538c2-kube-api-access-v9mb8\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:46.097750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:46.093627    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"]
Nov 03 18:13:46.100714 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:46.100674    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6cb8403-56ddc8cf9d-z66r9"]
Nov 03 18:13:46.929818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:46.929780    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbc9105-f7fb-4d68-a414-e850011538c2" path="/var/lib/kubelet/pods/7fbc9105-f7fb-4d68-a414-e850011538c2/volumes"
Nov 03 18:13:48.889093 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:48.889050    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main" containerID="cri-o://9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651" gracePeriod=45
Nov 03 18:13:49.189040 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.189018    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:13:49.276199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276168    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs\" (UniqueName: \"kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs\") pod \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") "
Nov 03 18:13:49.276356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276222    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir\") pod \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") "
Nov 03 18:13:49.276473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276453    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf\" (UniqueName: \"kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf\") pod \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") "
Nov 03 18:13:49.276530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276464    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir" pod "d98f3a96-0e02-4582-a7c3-574cbb2aed61" (UID: "d98f3a96-0e02-4582-a7c3-574cbb2aed61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.276573 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276527    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home\") pod \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") "
Nov 03 18:13:49.276616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276598    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm\") pod \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") "
Nov 03 18:13:49.276675 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276663    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache\") pod \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\" (UID: \"d98f3a96-0e02-4582-a7c3-574cbb2aed61\") "
Nov 03 18:13:49.276883 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.276856    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home" pod "d98f3a96-0e02-4582-a7c3-574cbb2aed61" (UID: "d98f3a96-0e02-4582-a7c3-574cbb2aed61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.277030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.277001    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache" pod "d98f3a96-0e02-4582-a7c3-574cbb2aed61" (UID: "d98f3a96-0e02-4582-a7c3-574cbb2aed61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.277351 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.277335    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.277430 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.277352    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.277430 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.277361    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.278863 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.278843    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm" pod "d98f3a96-0e02-4582-a7c3-574cbb2aed61" (UID: "d98f3a96-0e02-4582-a7c3-574cbb2aed61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:49.278950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.278933    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs" pod "d98f3a96-0e02-4582-a7c3-574cbb2aed61" (UID: "d98f3a96-0e02-4582-a7c3-574cbb2aed61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:49.279034 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.279014    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf" pod "d98f3a96-0e02-4582-a7c3-574cbb2aed61" (UID: "d98f3a96-0e02-4582-a7c3-574cbb2aed61"). InnerVolumeSpecName "kube-api-access-q6hbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:49.378786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.378745    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hbf\" (UniqueName: \"kubernetes.io/projected/d98f3a96-0e02-4582-a7c3-574cbb2aed61-kube-api-access-q6hbf\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.378786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.378785    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98f3a96-0e02-4582-a7c3-574cbb2aed61-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.379041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.378803    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98f3a96-0e02-4582-a7c3-574cbb2aed61-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:49.948743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.947805    6167 generic.go:356] "Generic (PLEG): container finished" podID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerID="9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651" exitCode=0
Nov 03 18:13:49.948743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.947903    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"
Nov 03 18:13:49.948743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.947964    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" event={"ID":"d98f3a96-0e02-4582-a7c3-574cbb2aed61","Type":"ContainerDied","Data":"9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651"}
Nov 03 18:13:49.948743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.948004    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb" event={"ID":"d98f3a96-0e02-4582-a7c3-574cbb2aed61","Type":"ContainerDied","Data":"c3b69673697b8a7753e18bce00385fe8d71f04faacf16c5c33d3b192e4a80511"}
Nov 03 18:13:49.948743 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.948026    6167 scope.go:122] "RemoveContainer" containerID="9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651"
Nov 03 18:13:49.973167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.970364    6167 scope.go:122] "RemoveContainer" containerID="9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651"
Nov 03 18:13:49.973167 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:49.970811    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651\": container with ID starting with 9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651 not found: ID does not exist" containerID="9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651"
Nov 03 18:13:49.973167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:49.970851    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651"} err="failed to get container status \"9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651\": rpc error: code = NotFound desc = could not find container \"9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651\": container with ID starting with 9fec50b1c91a304c7c062132556255b274ae6496f8940c4e6328e4f4878d9651 not found: ID does not exist"
Nov 03 18:13:50.115610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.115576    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"]
Nov 03 18:13:50.165844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.165804    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/simulator-w0-0ccbb9-kserve-d86bd4c6d-4lxcb"]
Nov 03 18:13:50.173830 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.173796    6167 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:13:50.173990 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.173867    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:13:50.723277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.723249    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:13:50.754664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.754637    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:13:50.818398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818363    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818540 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818438    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818540 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818480    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818540 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818521    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818592    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818647    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818697    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818752    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.818853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818799    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.818853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818834    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data\") pod \"0459156f-933e-4bb4-88bc-976d5e41c36f\" (UID: \"0459156f-933e-4bb4-88bc-976d5e41c36f\") "
Nov 03 18:13:50.819063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818880    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:50.819063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.818917    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.819063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.819031    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.819462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.819436    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.819667 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.819637    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0459156f-933e-4bb4-88bc-976d5e41c36f-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.819667 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.819662    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.819895 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.819702    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.819895 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.819751    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.821441 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.821413    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.821550 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.821531    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:50.821680 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.821657    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "kube-api-access-dk9m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:50.822118 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.822095    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo" pod "0459156f-933e-4bb4-88bc-976d5e41c36f" (UID: "0459156f-933e-4bb4-88bc-976d5e41c36f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:13:50.920786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.920732    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.920786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.920778    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.921440 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.920838    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.921440 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.921225    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:50.921553 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.921458    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.921786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.921763    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.921924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.921884    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.922023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.921926    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.922023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.921962    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.922178 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922027    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.922178 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922147    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data\") pod \"2716032b-1df8-4066-bff3-9ee3e2898296\" (UID: \"2716032b-1df8-4066-bff3-9ee3e2898296\") "
Nov 03 18:13:50.922444 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922293    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.922691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922667    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922758    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2716032b-1df8-4066-bff3-9ee3e2898296-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922775    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922788    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922802    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9m5\" (UniqueName: \"kubernetes.io/projected/0459156f-933e-4bb4-88bc-976d5e41c36f-kube-api-access-dk9m5\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922671    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922683    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.922844 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922816    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.923113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922860    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.923113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.922888    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0459156f-933e-4bb4-88bc-976d5e41c36f-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:50.923875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.923854    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "kube-api-access-s5pww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:50.924209 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.924189    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:13:50.924617 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.924588    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:50.925185 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.925166    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token" pod "2716032b-1df8-4066-bff3-9ee3e2898296" (UID: "2716032b-1df8-4066-bff3-9ee3e2898296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:50.931501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.931478    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" path="/var/lib/kubelet/pods/d98f3a96-0e02-4582-a7c3-574cbb2aed61/volumes"
Nov 03 18:13:50.964421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.964390    6167 generic.go:356] "Generic (PLEG): container finished" podID="2716032b-1df8-4066-bff3-9ee3e2898296" containerID="1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5" exitCode=0
Nov 03 18:13:50.964925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.964449    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"
Nov 03 18:13:50.964925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.964478    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" event={"ID":"2716032b-1df8-4066-bff3-9ee3e2898296","Type":"ContainerDied","Data":"1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5"}
Nov 03 18:13:50.964925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.964542    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4" event={"ID":"2716032b-1df8-4066-bff3-9ee3e2898296","Type":"ContainerDied","Data":"5936e4b55e46001d5eb7c8963b114614b5ad32d5c7740dc736b7147e8aec38b9"}
Nov 03 18:13:50.964925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.964575    6167 scope.go:122] "RemoveContainer" containerID="1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5"
Nov 03 18:13:50.968447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.968422    6167 generic.go:356] "Generic (PLEG): container finished" podID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerID="46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6" exitCode=0
Nov 03 18:13:50.968555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.968464    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"
Nov 03 18:13:50.968848 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.968463    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" event={"ID":"0459156f-933e-4bb4-88bc-976d5e41c36f","Type":"ContainerDied","Data":"46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6"}
Nov 03 18:13:50.968936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.968875    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb" event={"ID":"0459156f-933e-4bb4-88bc-976d5e41c36f","Type":"ContainerDied","Data":"fe84a07c55d9c3977c0e1b85a73ebff0af80dfb9fa20dc3bb521b5515cb873d6"}
Nov 03 18:13:50.982110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.982032    6167 scope.go:122] "RemoveContainer" containerID="1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5"
Nov 03 18:13:50.982573 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:50.982522    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5\": container with ID starting with 1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5 not found: ID does not exist" containerID="1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5"
Nov 03 18:13:50.982695 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.982574    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5"} err="failed to get container status \"1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5\": rpc error: code = NotFound desc = could not find container \"1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5\": container with ID starting with 1bad0af7d9c3232f5eb05b173c1e5380575d7673dca01092a5e105db97ed12d5 not found: ID does not exist"
Nov 03 18:13:50.982695 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.982598    6167 scope.go:122] "RemoveContainer" containerID="46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6"
Nov 03 18:13:50.996019 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.995984    6167 scope.go:122] "RemoveContainer" containerID="46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6"
Nov 03 18:13:50.996321 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:50.996294    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6\": container with ID starting with 46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6 not found: ID does not exist" containerID="46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6"
Nov 03 18:13:50.996388 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:50.996330    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6"} err="failed to get container status \"46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6\": rpc error: code = NotFound desc = could not find container \"46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6\": container with ID starting with 46d881635f814fdcb3113f2e5759211bd9d096635969091db68d927b9be14ab6 not found: ID does not exist"
Nov 03 18:13:51.024982 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.024957    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5pww\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-kube-api-access-s5pww\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:51.025106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.024985    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2716032b-1df8-4066-bff3-9ee3e2898296-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:51.025106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.024999    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:51.025106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.025011    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2716032b-1df8-4066-bff3-9ee3e2898296-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:51.025106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.025024    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:51.025106 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.025037    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2716032b-1df8-4066-bff3-9ee3e2898296-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:51.091668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.091637    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"]
Nov 03 18:13:51.095187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.095160    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-4kbq4"]
Nov 03 18:13:51.256276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.256196    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"]
Nov 03 18:13:51.259635 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.259615    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-qrgxb"]
Nov 03 18:13:51.580484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.580452    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"]
Nov 03 18:13:51.581696 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.580753    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" podUID="a00dea4a-0474-4f93-bfe9-d9b3046cc537" containerName="maas-api" containerID="cri-o://835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef" gracePeriod=30
Nov 03 18:13:51.903425 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.903394    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:13:51.985668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.981921    6167 generic.go:356] "Generic (PLEG): container finished" podID="a00dea4a-0474-4f93-bfe9-d9b3046cc537" containerID="835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef" exitCode=0
Nov 03 18:13:51.985668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.981979    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"
Nov 03 18:13:51.985668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.981973    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" event={"ID":"a00dea4a-0474-4f93-bfe9-d9b3046cc537","Type":"ContainerDied","Data":"835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef"}
Nov 03 18:13:51.985668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.982047    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6" event={"ID":"a00dea4a-0474-4f93-bfe9-d9b3046cc537","Type":"ContainerDied","Data":"cf3d172dbb16461045b09d1df82d9f11a042a07f19b393610f034180d8c3f370"}
Nov 03 18:13:51.985668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.982076    6167 scope.go:122] "RemoveContainer" containerID="835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef"
Nov 03 18:13:51.994558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.994532    6167 scope.go:122] "RemoveContainer" containerID="835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef"
Nov 03 18:13:51.994906 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:51.994883    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef\": container with ID starting with 835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef not found: ID does not exist" containerID="835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef"
Nov 03 18:13:51.995030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:51.994914    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef"} err="failed to get container status \"835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef\": rpc error: code = NotFound desc = could not find container \"835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef\": container with ID starting with 835e3be4ff9502c910086162e3294eda9efc160ab73d795a40e5f8984f2efeef not found: ID does not exist"
Nov 03 18:13:52.040062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.040033    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs\") pod \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") "
Nov 03 18:13:52.040254 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.040238    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj\" (UniqueName: \"kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj\") pod \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") "
Nov 03 18:13:52.040339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.040284    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\") pod \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") "
Nov 03 18:13:52.040432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.040407    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca\") pod \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\" (UID: \"a00dea4a-0474-4f93-bfe9-d9b3046cc537\") "
Nov 03 18:13:52.040945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.040912    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca" pod "a00dea4a-0474-4f93-bfe9-d9b3046cc537" (UID: "a00dea4a-0474-4f93-bfe9-d9b3046cc537"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:52.041142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.041117    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00dea4a-0474-4f93-bfe9-d9b3046cc537-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:52.043080 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.043051    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls" pod "a00dea4a-0474-4f93-bfe9-d9b3046cc537" (UID: "a00dea4a-0474-4f93-bfe9-d9b3046cc537"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:52.043199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.043181    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj" pod "a00dea4a-0474-4f93-bfe9-d9b3046cc537" (UID: "a00dea4a-0474-4f93-bfe9-d9b3046cc537"). InnerVolumeSpecName "kube-api-access-2zkpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:52.043364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.043334    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs" pod "a00dea4a-0474-4f93-bfe9-d9b3046cc537" (UID: "a00dea4a-0474-4f93-bfe9-d9b3046cc537"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:52.156798 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.145169    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zkpj\" (UniqueName: \"kubernetes.io/projected/a00dea4a-0474-4f93-bfe9-d9b3046cc537-kube-api-access-2zkpj\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:52.156798 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.145215    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:52.156798 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.145230    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00dea4a-0474-4f93-bfe9-d9b3046cc537-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:13:52.405064 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.405032    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"]
Nov 03 18:13:52.455408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.455332    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c91bee-cbb867f9c-nztz6"]
Nov 03 18:13:52.929816 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.929769    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" path="/var/lib/kubelet/pods/0459156f-933e-4bb4-88bc-976d5e41c36f/volumes"
Nov 03 18:13:52.930360 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.930342    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" path="/var/lib/kubelet/pods/2716032b-1df8-4066-bff3-9ee3e2898296/volumes"
Nov 03 18:13:52.930808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:13:52.930794    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00dea4a-0474-4f93-bfe9-d9b3046cc537" path="/var/lib/kubelet/pods/a00dea4a-0474-4f93-bfe9-d9b3046cc537/volumes"
Nov 03 18:13:54.522852 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:13:54.522813    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:14:01.382650 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:01.382617    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"]
Nov 03 18:14:11.670925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:11.670888    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"]
Nov 03 18:14:11.721484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:11.721449    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"]
Nov 03 18:14:16.404690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.404647    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main" containerID="cri-o://9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd" gracePeriod=45
Nov 03 18:14:16.877412 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.877382    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:14:16.962071 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962037    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache\") pod \"08886127-238f-4161-b547-897e45db3c05\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") "
Nov 03 18:14:16.962256 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962102    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w\" (UniqueName: \"kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w\") pod \"08886127-238f-4161-b547-897e45db3c05\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") "
Nov 03 18:14:16.962256 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962234    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs\" (UniqueName: \"kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs\") pod \"08886127-238f-4161-b547-897e45db3c05\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") "
Nov 03 18:14:16.962355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962273    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm\") pod \"08886127-238f-4161-b547-897e45db3c05\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") "
Nov 03 18:14:16.962410 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962351    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache" pod "08886127-238f-4161-b547-897e45db3c05" (UID: "08886127-238f-4161-b547-897e45db3c05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:16.962410 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962394    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home\") pod \"08886127-238f-4161-b547-897e45db3c05\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") "
Nov 03 18:14:16.962522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962442    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir\") pod \"08886127-238f-4161-b547-897e45db3c05\" (UID: \"08886127-238f-4161-b547-897e45db3c05\") "
Nov 03 18:14:16.962685 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962662    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home" pod "08886127-238f-4161-b547-897e45db3c05" (UID: "08886127-238f-4161-b547-897e45db3c05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:16.962948 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.962920    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir" pod "08886127-238f-4161-b547-897e45db3c05" (UID: "08886127-238f-4161-b547-897e45db3c05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:16.963152 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.963134    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:16.963226 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.963155    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:16.963226 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.963170    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:16.964627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.964598    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs" pod "08886127-238f-4161-b547-897e45db3c05" (UID: "08886127-238f-4161-b547-897e45db3c05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:16.964835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.964814    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w" pod "08886127-238f-4161-b547-897e45db3c05" (UID: "08886127-238f-4161-b547-897e45db3c05"). InnerVolumeSpecName "kube-api-access-czh9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:16.964934 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:16.964910    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm" pod "08886127-238f-4161-b547-897e45db3c05" (UID: "08886127-238f-4161-b547-897e45db3c05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:17.064703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.064668    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czh9w\" (UniqueName: \"kubernetes.io/projected/08886127-238f-4161-b547-897e45db3c05-kube-api-access-czh9w\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.064703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.064700    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08886127-238f-4161-b547-897e45db3c05-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.064918 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.064714    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08886127-238f-4161-b547-897e45db3c05-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:17.253556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.253470    6167 generic.go:356] "Generic (PLEG): container finished" podID="08886127-238f-4161-b547-897e45db3c05" containerID="9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd" exitCode=0
Nov 03 18:14:17.253556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.253513    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"
Nov 03 18:14:17.253556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.253530    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" event={"ID":"08886127-238f-4161-b547-897e45db3c05","Type":"ContainerDied","Data":"9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd"}
Nov 03 18:14:17.253796 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.253573    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5" event={"ID":"08886127-238f-4161-b547-897e45db3c05","Type":"ContainerDied","Data":"4fe00b8d04bb733b4ac8efb7253b6358fb01b80ecb870d819f947c6d3476841a"}
Nov 03 18:14:17.253796 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.253594    6167 scope.go:122] "RemoveContainer" containerID="9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd"
Nov 03 18:14:17.274594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.274567    6167 scope.go:122] "RemoveContainer" containerID="9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd"
Nov 03 18:14:17.275126 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:17.275045    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd\": container with ID starting with 9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd not found: ID does not exist" containerID="9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd"
Nov 03 18:14:17.275126 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.275091    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd"} err="failed to get container status \"9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd\": rpc error: code = NotFound desc = could not find container \"9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd\": container with ID starting with 9414662b543c5439e86ea0ecf11b93d44ecd40a94bb6bd0171046d883f860bdd not found: ID does not exist"
Nov 03 18:14:17.423206 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.423168    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"]
Nov 03 18:14:17.475456 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:17.475419    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/premium-w3-0adcbc-kserve-5d4c6bffcb-7xft5"]
Nov 03 18:14:18.929622 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:18.929593    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08886127-238f-4161-b547-897e45db3c05" path="/var/lib/kubelet/pods/08886127-238f-4161-b547-897e45db3c05/volumes"
Nov 03 18:14:24.527423 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:24.527352    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:14:26.693521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:26.693460    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" containerID="cri-o://ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce" gracePeriod=45
Nov 03 18:14:26.743005 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:26.742950    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" containerID="cri-o://1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f" gracePeriod=45
Nov 03 18:14:26.826823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:26.826791    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused"
Nov 03 18:14:26.882630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:26.882587    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused"
Nov 03 18:14:27.099701 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.099667    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:14:27.103597 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.103570    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:14:27.238394 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238317    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs\") pod \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") "
Nov 03 18:14:27.238394 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238359    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache\") pod \"73a3c197-2658-465b-9271-d7367e9e79e9\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") "
Nov 03 18:14:27.238620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238421    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache\") pod \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") "
Nov 03 18:14:27.238620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238533    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home\") pod \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") "
Nov 03 18:14:27.238620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238567    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs\") pod \"73a3c197-2658-465b-9271-d7367e9e79e9\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") "
Nov 03 18:14:27.238620 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238606    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm\") pod \"73a3c197-2658-465b-9271-d7367e9e79e9\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") "
Nov 03 18:14:27.238902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238641    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir\") pod \"73a3c197-2658-465b-9271-d7367e9e79e9\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") "
Nov 03 18:14:27.238902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238674    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir\") pod \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") "
Nov 03 18:14:27.238902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238672    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache" pod "73a3c197-2658-465b-9271-d7367e9e79e9" (UID: "73a3c197-2658-465b-9271-d7367e9e79e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.238902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238771    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache" pod "cfb1bcf2-cf99-4255-933f-48a10b71f9ba" (UID: "cfb1bcf2-cf99-4255-933f-48a10b71f9ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.238902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238774    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h\" (UniqueName: \"kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h\") pod \"73a3c197-2658-465b-9271-d7367e9e79e9\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") "
Nov 03 18:14:27.238902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.238866    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz\" (UniqueName: \"kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz\") pod \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") "
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.239023    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home\") pod \"73a3c197-2658-465b-9271-d7367e9e79e9\" (UID: \"73a3c197-2658-465b-9271-d7367e9e79e9\") "
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.239059    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm\") pod \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\" (UID: \"cfb1bcf2-cf99-4255-933f-48a10b71f9ba\") "
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.239736    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home" pod "73a3c197-2658-465b-9271-d7367e9e79e9" (UID: "73a3c197-2658-465b-9271-d7367e9e79e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.239932    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home" pod "cfb1bcf2-cf99-4255-933f-48a10b71f9ba" (UID: "cfb1bcf2-cf99-4255-933f-48a10b71f9ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.239947    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.239984    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.240001    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.240218    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir" pod "73a3c197-2658-465b-9271-d7367e9e79e9" (UID: "73a3c197-2658-465b-9271-d7367e9e79e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.240610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.240275    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir" pod "cfb1bcf2-cf99-4255-933f-48a10b71f9ba" (UID: "cfb1bcf2-cf99-4255-933f-48a10b71f9ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.243086 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.242404    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm" pod "cfb1bcf2-cf99-4255-933f-48a10b71f9ba" (UID: "cfb1bcf2-cf99-4255-933f-48a10b71f9ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.243570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.243536    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm" pod "73a3c197-2658-465b-9271-d7367e9e79e9" (UID: "73a3c197-2658-465b-9271-d7367e9e79e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:27.243827 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.243800    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs" pod "cfb1bcf2-cf99-4255-933f-48a10b71f9ba" (UID: "cfb1bcf2-cf99-4255-933f-48a10b71f9ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:27.245281 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.245255    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz" pod "cfb1bcf2-cf99-4255-933f-48a10b71f9ba" (UID: "cfb1bcf2-cf99-4255-933f-48a10b71f9ba"). InnerVolumeSpecName "kube-api-access-pjjzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:27.246102 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.246081    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs" pod "73a3c197-2658-465b-9271-d7367e9e79e9" (UID: "73a3c197-2658-465b-9271-d7367e9e79e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:27.246183 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.246079    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h" pod "73a3c197-2658-465b-9271-d7367e9e79e9" (UID: "73a3c197-2658-465b-9271-d7367e9e79e9"). InnerVolumeSpecName "kube-api-access-nsj7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348283    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348329    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348342    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348363    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a3c197-2658-465b-9271-d7367e9e79e9-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348376    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348389    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a3c197-2658-465b-9271-d7367e9e79e9-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348402    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348416    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj7h\" (UniqueName: \"kubernetes.io/projected/73a3c197-2658-465b-9271-d7367e9e79e9-kube-api-access-nsj7h\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.348789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.348431    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjzz\" (UniqueName: \"kubernetes.io/projected/cfb1bcf2-cf99-4255-933f-48a10b71f9ba-kube-api-access-pjjzz\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:27.369415 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.369376    6167 generic.go:356] "Generic (PLEG): container finished" podID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerID="1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f" exitCode=0
Nov 03 18:14:27.369595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.369441    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"
Nov 03 18:14:27.369595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.369486    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" event={"ID":"cfb1bcf2-cf99-4255-933f-48a10b71f9ba","Type":"ContainerDied","Data":"1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f"}
Nov 03 18:14:27.369595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.369526    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg" event={"ID":"cfb1bcf2-cf99-4255-933f-48a10b71f9ba","Type":"ContainerDied","Data":"6f2783db5b0c7be9f79532a64d98ba0793135c48baabfa2d042809069e26045c"}
Nov 03 18:14:27.369595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.369551    6167 scope.go:122] "RemoveContainer" containerID="1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f"
Nov 03 18:14:27.377752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.374031    6167 generic.go:356] "Generic (PLEG): container finished" podID="73a3c197-2658-465b-9271-d7367e9e79e9" containerID="ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce" exitCode=0
Nov 03 18:14:27.377752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.374145    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" event={"ID":"73a3c197-2658-465b-9271-d7367e9e79e9","Type":"ContainerDied","Data":"ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce"}
Nov 03 18:14:27.377752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.374175    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk" event={"ID":"73a3c197-2658-465b-9271-d7367e9e79e9","Type":"ContainerDied","Data":"781117bb64ea1b6c2b879e41251ed568b3a859133be5ef41cc9957768b10f9ab"}
Nov 03 18:14:27.377752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.374239    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"
Nov 03 18:14:27.393226 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.392687    6167 scope.go:122] "RemoveContainer" containerID="1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f"
Nov 03 18:14:27.393515 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:27.393483    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f\": container with ID starting with 1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f not found: ID does not exist" containerID="1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f"
Nov 03 18:14:27.393589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.393530    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f"} err="failed to get container status \"1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f\": rpc error: code = NotFound desc = could not find container \"1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f\": container with ID starting with 1522eb5cd32a376ba4893744f0bca5c0e1b6e829da30e9c792dfb300e0f3b76f not found: ID does not exist"
Nov 03 18:14:27.393589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.393557    6167 scope.go:122] "RemoveContainer" containerID="ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce"
Nov 03 18:14:27.419837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.418441    6167 scope.go:122] "RemoveContainer" containerID="ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce"
Nov 03 18:14:27.429228 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:27.429192    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce\": container with ID starting with ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce not found: ID does not exist" containerID="ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce"
Nov 03 18:14:27.429420 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.429238    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce"} err="failed to get container status \"ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce\": rpc error: code = NotFound desc = could not find container \"ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce\": container with ID starting with ab9e3625a17c6ee392c486c1a0068d288b0103549edd93b4bb63a5f522fff1ce not found: ID does not exist"
Nov 03 18:14:27.554441 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.554355    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"]
Nov 03 18:14:27.560015 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.559976    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-w6-aab753-kserve-6fc576b9b6-c29hg"]
Nov 03 18:14:27.664520 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.664483    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"]
Nov 03 18:14:27.668044 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:27.668019    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aab753/distinct-2-w6-aab753-kserve-6b69b9c7c-xlblk"]
Nov 03 18:14:28.930848 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:28.930776    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" path="/var/lib/kubelet/pods/73a3c197-2658-465b-9271-d7367e9e79e9/volumes"
Nov 03 18:14:28.931309 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:28.931294    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" path="/var/lib/kubelet/pods/cfb1bcf2-cf99-4255-933f-48a10b71f9ba/volumes"
Nov 03 18:14:33.127014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.126980    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"]
Nov 03 18:14:33.127751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127710    6167 cpu_manager.go:414] "containerMap: removing container" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main"
Nov 03 18:14:33.127751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127738    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127757    6167 cpu_manager.go:414] "containerMap: removing container" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127762    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127774    6167 cpu_manager.go:414] "containerMap: removing container" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127779    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127787    6167 cpu_manager.go:414] "containerMap: removing container" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127793    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127812    6167 cpu_manager.go:414] "containerMap: removing container" podUID="7fbc9105-f7fb-4d68-a414-e850011538c2" containerName="maas-api"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127816    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbc9105-f7fb-4d68-a414-e850011538c2" containerName="maas-api"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127825    6167 cpu_manager.go:414] "containerMap: removing container" podUID="a00dea4a-0474-4f93-bfe9-d9b3046cc537" containerName="maas-api"
Nov 03 18:14:33.127837 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127831    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00dea4a-0474-4f93-bfe9-d9b3046cc537" containerName="maas-api"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127842    6167 cpu_manager.go:414] "containerMap: removing container" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127847    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127858    6167 cpu_manager.go:414] "containerMap: removing container" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127863    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127977    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbc9105-f7fb-4d68-a414-e850011538c2" containerName="maas-api"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127984    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00dea4a-0474-4f93-bfe9-d9b3046cc537" containerName="maas-api"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127991    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98f3a96-0e02-4582-a7c3-574cbb2aed61" containerName="main"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.127999    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="0459156f-933e-4bb4-88bc-976d5e41c36f" containerName="istio-proxy"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.128009    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a3c197-2658-465b-9271-d7367e9e79e9" containerName="main"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.128017    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb1bcf2-cf99-4255-933f-48a10b71f9ba" containerName="main"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.128023    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="08886127-238f-4161-b547-897e45db3c05" containerName="main"
Nov 03 18:14:33.128132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.128029    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="2716032b-1df8-4066-bff3-9ee3e2898296" containerName="istio-proxy"
Nov 03 18:14:33.132932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.132914    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.186171 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.186136    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1720c00c-openshift-default-dockercfg-ccffc\""
Nov 03 18:14:33.235970 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.235927    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2mf\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236159 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.235990    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236159 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236131    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236161    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236350 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236306    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236413 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236352    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236443    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236604 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236582    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.236672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.236636    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.240217 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.240193    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"]
Nov 03 18:14:33.338554 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338517    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.338771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338571    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.338771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338602    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.338771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338734    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.338963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338778    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.338963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338842    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2mf\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.338963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338885    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.338982    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.339007    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.339060    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.339092    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339320 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.339126    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339377 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.339347    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.339592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.339566    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.341044 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.341019    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.341483 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.341460    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.400140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.400082    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.400265 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.400241    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2mf\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf\") pod \"e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") " pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.449925 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.449900    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:33.871753 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:14:33.871687    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b7f3f_5a8f_434b_aa34_8bc6d64ee01f.slice/crio-298e8ac5167569d2609e7e9850dd2cc5d8597f482d1258dd6b4ebd95999758c0 WatchSource:0}: Error finding container 298e8ac5167569d2609e7e9850dd2cc5d8597f482d1258dd6b4ebd95999758c0: Status 404 returned error can't find the container with id 298e8ac5167569d2609e7e9850dd2cc5d8597f482d1258dd6b4ebd95999758c0
Nov 03 18:14:33.874894 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.874854    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:14:33.874986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.874943    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:14:33.875022 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.874987    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:14:33.877082 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:33.877054    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"]
Nov 03 18:14:34.448930 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:34.448893    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" event={"ID":"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f","Type":"ContainerStarted","Data":"bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da"}
Nov 03 18:14:34.449570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:34.449527    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" event={"ID":"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f","Type":"ContainerStarted","Data":"298e8ac5167569d2609e7e9850dd2cc5d8597f482d1258dd6b4ebd95999758c0"}
Nov 03 18:14:34.451263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:34.451242    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:34.454658 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:34.454632    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 18:14:34.454866 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:34.454843    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 18:14:34.574563 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:34.574508    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podStartSLOduration=1.574492825 podStartE2EDuration="1.574492825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:34.572599693 +0000 UTC m=+1729.802924766" watchObservedRunningTime="2026-11-03 18:14:34.574492825 +0000 UTC m=+1729.804817896"
Nov 03 18:14:35.455848 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:35.455810    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 18:14:35.456391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:35.455871    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 18:14:36.454063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:36.454027    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 18:14:36.454455 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:36.454335    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 18:14:37.455136 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:37.455091    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 18:14:37.455614 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:37.455165    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 18:14:38.451119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:38.451078    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 18:14:38.451452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:38.451412    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 18:14:39.452742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:39.452094    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 18:14:39.452742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:39.452158    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 18:14:40.454937 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:40.454908    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:40.455405 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:40.455384    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:40.456046 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:40.456023    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:14:41.288452 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.288421    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"]
Nov 03 18:14:41.292660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.292643    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.347076 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.347045    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c1b2d2-serving-cert\""
Nov 03 18:14:41.401920 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.401881    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"]
Nov 03 18:14:41.412110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.412079    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpv4\" (UniqueName: \"kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.412382 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.412362    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.412643 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.412627    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.412824 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.412808    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.514169 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.514132    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.514668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.514265    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpv4\" (UniqueName: \"kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.514668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.514355    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.514668 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:41.514486    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-serving-cert: secret "maas-api-e2e-shared-b-w2-c1b2d2-serving-cert" not found
Nov 03 18:14:41.514668 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.514509    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.514668 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:41.514564    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls podName:d1a2a820-3ce2-4afb-808e-f8378d7540ed nodeName:}" failed. No retries permitted until 2026-11-03 18:14:42.014544537 +0000 UTC m=+1737.244869604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" (UID: "d1a2a820-3ce2-4afb-808e-f8378d7540ed") : secret "maas-api-e2e-shared-b-w2-c1b2d2-serving-cert" not found
Nov 03 18:14:41.515037 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.515004    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.539542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.539456    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:41.619874 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:41.619837    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpv4\" (UniqueName: \"kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:42.022332 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:42.022294    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:42.024877 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:42.024854    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:42.206495 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:42.206462    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:42.592874 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:42.592847    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"]
Nov 03 18:14:43.524923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:43.524884    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" event={"ID":"d1a2a820-3ce2-4afb-808e-f8378d7540ed","Type":"ContainerStarted","Data":"360ed44c4d06f00a253176b973bef14156e28a2b378123589560e7ffff91c0e4"}
Nov 03 18:14:44.533895 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:44.533859    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" event={"ID":"d1a2a820-3ce2-4afb-808e-f8378d7540ed","Type":"ContainerStarted","Data":"2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6"}
Nov 03 18:14:44.534403 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:44.533957    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:44.653513 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:44.653460    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" podStartSLOduration=2.715545803 podStartE2EDuration="3.653444584s" totalImagesPullingTime="937.898781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:44.651956994 +0000 UTC m=+1739.882282065" watchObservedRunningTime="2026-11-03 18:14:44.653444584 +0000 UTC m=+1739.883769677"
Nov 03 18:14:46.400901 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:46.400867    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"]
Nov 03 18:14:46.401395 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:46.401132    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy" containerID="cri-o://5499979f58aeb86b4e632ea27a7e97e5bef2a4df39ae3c1dc8b2b620c4be22c8" gracePeriod=30
Nov 03 18:14:50.385286 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:50.385252    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:14:50.563266 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:50.563235    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:14:51.561770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:51.561686    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"]
Nov 03 18:14:51.562289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:51.561971    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" podUID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" containerName="maas-api" containerID="cri-o://8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8" gracePeriod=30
Nov 03 18:14:51.629897 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:51.628044    6167 generic.go:356] "Generic (PLEG): container finished" podID="cb280431-6efb-4ee4-8446-ef91773809b3" containerID="5499979f58aeb86b4e632ea27a7e97e5bef2a4df39ae3c1dc8b2b620c4be22c8" exitCode=0
Nov 03 18:14:51.629897 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:51.628204    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" event={"ID":"cb280431-6efb-4ee4-8446-ef91773809b3","Type":"ContainerDied","Data":"5499979f58aeb86b4e632ea27a7e97e5bef2a4df39ae3c1dc8b2b620c4be22c8"}
Nov 03 18:14:51.789703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:51.789672    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:14:51.979559 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:51.979532    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.029917    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030023    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030100    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030176    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030215    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030262    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030309    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030346    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.030418    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert\") pod \"cb280431-6efb-4ee4-8446-ef91773809b3\" (UID: \"cb280431-6efb-4ee4-8446-ef91773809b3\") "
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.041832    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.042086    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.042367    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.044740 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.043287    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.057029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.056986    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:52.063531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.063490    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:52.063781 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.063684    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "kube-api-access-qjqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.064447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.064411    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:52.068442 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.066525    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token" pod "cb280431-6efb-4ee4-8446-ef91773809b3" (UID: "cb280431-6efb-4ee4-8446-ef91773809b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.132153    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699\" (UniqueName: \"kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699\") pod \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") "
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.132228    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs\") pod \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") "
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.132445    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca\") pod \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") "
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.132494    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\") pod \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\" (UID: \"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1\") "
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.133895    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.133923    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.133939    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.133955    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.133971    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.133985    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb280431-6efb-4ee4-8446-ef91773809b3-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.134000    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjqgv\" (UniqueName: \"kubernetes.io/projected/cb280431-6efb-4ee4-8446-ef91773809b3-kube-api-access-qjqgv\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.134016    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb280431-6efb-4ee4-8446-ef91773809b3-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.134655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.134030    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb280431-6efb-4ee4-8446-ef91773809b3-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.136111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.135807    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca" pod "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" (UID: "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:52.136659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.136620    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs" pod "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" (UID: "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:52.138971 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.138927    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699" pod "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" (UID: "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1"). InnerVolumeSpecName "kube-api-access-tf699". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:52.140950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.140917    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls" pod "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" (UID: "7ca50e99-73fa-4476-9fe3-abcdcc4b92a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:52.235792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.235652    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.235792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.235694    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.235792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.235709    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf699\" (UniqueName: \"kubernetes.io/projected/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-kube-api-access-tf699\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.235792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.235749    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:14:52.677160 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.666912    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7" event={"ID":"cb280431-6efb-4ee4-8446-ef91773809b3","Type":"ContainerDied","Data":"d728ff7fcf5481ac829443419d84e8cd5d651544e5b6714d3c396a66db549570"}
Nov 03 18:14:52.677160 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.666985    6167 scope.go:122] "RemoveContainer" containerID="5499979f58aeb86b4e632ea27a7e97e5bef2a4df39ae3c1dc8b2b620c4be22c8"
Nov 03 18:14:52.677160 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.667131    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"
Nov 03 18:14:52.682189 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.679602    6167 generic.go:356] "Generic (PLEG): container finished" podID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" containerID="8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8" exitCode=0
Nov 03 18:14:52.706583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.688912    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" event={"ID":"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1","Type":"ContainerDied","Data":"8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8"}
Nov 03 18:14:52.706583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.688988    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" event={"ID":"7ca50e99-73fa-4476-9fe3-abcdcc4b92a1","Type":"ContainerDied","Data":"f2e2917aed7978da0ad22634275655075e2d3e51b7e80ccccfc87e9d9d3b711f"}
Nov 03 18:14:52.706583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.689073    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"
Nov 03 18:14:52.817824 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.817230    6167 scope.go:122] "RemoveContainer" containerID="8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8"
Nov 03 18:14:52.853757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.829581    6167 scope.go:122] "RemoveContainer" containerID="8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8"
Nov 03 18:14:52.853757 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:52.852231    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8\": container with ID starting with 8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8 not found: ID does not exist" containerID="8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8"
Nov 03 18:14:52.853757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.852273    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8"} err="failed to get container status \"8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8\": rpc error: code = NotFound desc = could not find container \"8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8\": container with ID starting with 8fd07b8b85eefacb13142cacbe05a815c384e2e7a3d81478f087c16d43a59eb8 not found: ID does not exist"
Nov 03 18:14:52.997792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:52.986228    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"]
Nov 03 18:14:53.015045 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:53.015006    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-c6hz7"]
Nov 03 18:14:53.101910 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:53.101868    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"]
Nov 03 18:14:53.107054 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:53.107020    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj"]
Nov 03 18:14:54.526067 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:14:54.526022    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:14:54.801909 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:54.801808    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-260dd0-857ccb74d5-fjvxj" podUID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.115:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:14:54.930809 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:54.930774    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" path="/var/lib/kubelet/pods/7ca50e99-73fa-4476-9fe3-abcdcc4b92a1/volumes"
Nov 03 18:14:54.931295 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:54.931281    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" path="/var/lib/kubelet/pods/cb280431-6efb-4ee4-8446-ef91773809b3/volumes"
Nov 03 18:14:55.909976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:55.909936    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"]
Nov 03 18:14:55.910590 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:55.910175    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" containerID="cri-o://a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722" gracePeriod=30
Nov 03 18:14:55.913967 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:55.913939    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"]
Nov 03 18:14:55.914180 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:55.914157    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" containerID="cri-o://cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f" gracePeriod=30
Nov 03 18:14:55.921267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:55.921241    6167 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" secret="" err="secret \"e2e-worker-w0-0ccbb9-openshift-default-dockercfg-8dgdl\" not found"
Nov 03 18:14:55.925242 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:55.925221    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"]
Nov 03 18:14:56.726359 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:56.726323    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy" containerID="cri-o://db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf" gracePeriod=30
Nov 03 18:14:58.846795 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:58.846762    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:58.847190 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:58.846814    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:59.985586 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:59.985546    6167 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:59.985995 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:14:59.985602    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:01.398790 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.397707    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:15:01.408056 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.408031    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:15:01.465697 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.465654    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.465893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.465709    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.465893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.465772    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.465893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.465816    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.465890    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.465972    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466012    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466097    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466140    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466135    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.466300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466172    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466204    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466511 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466378    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466511 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466424    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466511 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466479    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy\") pod \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\" (UID: \"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb\") "
Nov 03 18:15:01.466677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466522    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466571    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.466677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466614    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466652    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466676    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy\") pod \"3cd797ae-f45d-4466-ab72-fe20b4af183c\" (UID: \"3cd797ae-f45d-4466-ab72-fe20b4af183c\") "
Nov 03 18:15:01.466959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466777    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.466959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466806    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.466959 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.466926    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.467353 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467306    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.467617 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467581    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.467617 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467614    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.467801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467630    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.467801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467644    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.467801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467656    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.467801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.467669    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.468092 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.468065    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.468466 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.468443    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:01.469014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.468989    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.469199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.469179    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.469286 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.469269    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "kube-api-access-xvssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.473172 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.472805    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:01.474111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.473637    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "kube-api-access-v7mfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.474111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.473874    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:01.474111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.474072    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.474874 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.474831    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.475091 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.475064    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy" pod "3cd797ae-f45d-4466-ab72-fe20b4af183c" (UID: "3cd797ae-f45d-4466-ab72-fe20b4af183c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:01.475515 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.475490    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token" pod "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" (UID: "3653f1be-c7ce-41bf-a5c8-1fdd40762aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.569646 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569602    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569646 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569642    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569657    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd797ae-f45d-4466-ab72-fe20b4af183c-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569672    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569686    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569699    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569713    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569750    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569765    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mfw\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-kube-api-access-v7mfw\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569781    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvssn\" (UniqueName: \"kubernetes.io/projected/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb-kube-api-access-xvssn\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569795    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd797ae-f45d-4466-ab72-fe20b4af183c-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.569926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.569808    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd797ae-f45d-4466-ab72-fe20b4af183c-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.771248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.771210    6167 generic.go:356] "Generic (PLEG): container finished" podID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerID="a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722" exitCode=0
Nov 03 18:15:01.771418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.771287    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"
Nov 03 18:15:01.771418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.771297    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" event={"ID":"3cd797ae-f45d-4466-ab72-fe20b4af183c","Type":"ContainerDied","Data":"a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722"}
Nov 03 18:15:01.771418 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.771401    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr" event={"ID":"3cd797ae-f45d-4466-ab72-fe20b4af183c","Type":"ContainerDied","Data":"47bbe47d78544adbd66acef0c8337a94494d9a79320cbb0378e51fed6a15d914"}
Nov 03 18:15:01.771586 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.771433    6167 scope.go:122] "RemoveContainer" containerID="a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722"
Nov 03 18:15:01.773578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.773554    6167 generic.go:356] "Generic (PLEG): container finished" podID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerID="cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f" exitCode=0
Nov 03 18:15:01.773778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.773605    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"
Nov 03 18:15:01.775264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.773617    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" event={"ID":"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb","Type":"ContainerDied","Data":"cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f"}
Nov 03 18:15:01.775264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.773884    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575" event={"ID":"3653f1be-c7ce-41bf-a5c8-1fdd40762aeb","Type":"ContainerDied","Data":"cb6551511042eeaff1d09fdab40f225fe7bdf9ff9290bec6eedf1a44bd52ab13"}
Nov 03 18:15:01.842683 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.842661    6167 scope.go:122] "RemoveContainer" containerID="a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722"
Nov 03 18:15:01.843058 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:01.843029    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722\": container with ID starting with a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722 not found: ID does not exist" containerID="a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722"
Nov 03 18:15:01.843165 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.843071    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722"} err="failed to get container status \"a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722\": rpc error: code = NotFound desc = could not find container \"a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722\": container with ID starting with a8dbf2dad8603e430132bc6ae8462efc4381d8fc3f8ce267e4879eb5bef94722 not found: ID does not exist"
Nov 03 18:15:01.843165 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.843095    6167 scope.go:122] "RemoveContainer" containerID="cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f"
Nov 03 18:15:01.855857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.855835    6167 scope.go:122] "RemoveContainer" containerID="cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f"
Nov 03 18:15:01.856240 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:01.856213    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f\": container with ID starting with cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f not found: ID does not exist" containerID="cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f"
Nov 03 18:15:01.856342 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.856251    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f"} err="failed to get container status \"cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f\": rpc error: code = NotFound desc = could not find container \"cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f\": container with ID starting with cf01399baa3cc42d898af24288961ef475c3631c395cb61226c27d5a578a7d1f not found: ID does not exist"
Nov 03 18:15:01.959838 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.957155    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"]
Nov 03 18:15:01.962567 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:01.962546    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-wp575"]
Nov 03 18:15:02.020815 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.020777    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 18:15:02.075280 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.075246    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"]
Nov 03 18:15:02.079001 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.078973    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-7qzqr"]
Nov 03 18:15:02.179243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179151    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179194    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179243 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179235    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179255    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179310    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179383    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179445    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179470    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.179913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.179579    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9\") pod \"21faf221-c045-4d19-88ca-6a2fd72667be\" (UID: \"21faf221-c045-4d19-88ca-6a2fd72667be\") "
Nov 03 18:15:02.181612 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.181572    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:02.182618 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.182419    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:02.183151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.182929    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:02.183151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.182965    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "kube-api-access-cm2t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:02.183151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.183087    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:02.183379 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.183297    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:02.183467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.183448    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:02.183986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.183968    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:02.185266 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.185231    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo" pod "21faf221-c045-4d19-88ca-6a2fd72667be" (UID: "21faf221-c045-4d19-88ca-6a2fd72667be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:02.281774 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281696    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2t9\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-kube-api-access-cm2t9\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.281774 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281768    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21faf221-c045-4d19-88ca-6a2fd72667be-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.281774 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281780    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.282088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281789    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.282088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281798    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.282088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281807    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21faf221-c045-4d19-88ca-6a2fd72667be-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.282088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281815    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21faf221-c045-4d19-88ca-6a2fd72667be-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.282088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281825    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.282088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.281844    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21faf221-c045-4d19-88ca-6a2fd72667be-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:02.785050 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.785018    6167 generic.go:356] "Generic (PLEG): container finished" podID="21faf221-c045-4d19-88ca-6a2fd72667be" containerID="db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf" exitCode=0
Nov 03 18:15:02.785522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.785069    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"
Nov 03 18:15:02.785522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.785104    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" event={"ID":"21faf221-c045-4d19-88ca-6a2fd72667be","Type":"ContainerDied","Data":"db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf"}
Nov 03 18:15:02.785522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.785150    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs" event={"ID":"21faf221-c045-4d19-88ca-6a2fd72667be","Type":"ContainerDied","Data":"d1c6df7f891950ce28738da1442dbfef293e4cfebad71c169424ce325d92fc77"}
Nov 03 18:15:02.785522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.785175    6167 scope.go:122] "RemoveContainer" containerID="db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf"
Nov 03 18:15:02.797031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.797010    6167 scope.go:122] "RemoveContainer" containerID="d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"
Nov 03 18:15:02.807187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.807168    6167 scope.go:122] "RemoveContainer" containerID="db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf"
Nov 03 18:15:02.807483 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:02.807464    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf\": container with ID starting with db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf not found: ID does not exist" containerID="db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf"
Nov 03 18:15:02.807528 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.807491    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf"} err="failed to get container status \"db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf\": rpc error: code = NotFound desc = could not find container \"db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf\": container with ID starting with db37e36d686dfe6b0f75b23c7f6151520bbee349f18149ed8b7fbb5b90cb24cf not found: ID does not exist"
Nov 03 18:15:02.807528 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.807515    6167 scope.go:122] "RemoveContainer" containerID="d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"
Nov 03 18:15:02.807773 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:02.807755    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a\": container with ID starting with d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a not found: ID does not exist" containerID="d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"
Nov 03 18:15:02.807831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.807780    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a"} err="failed to get container status \"d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a\": rpc error: code = NotFound desc = could not find container \"d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a\": container with ID starting with d78b30f909d2137319a3a9524245febfd3dc7722fa2c3f6692a900ab6a45009a not found: ID does not exist"
Nov 03 18:15:02.929548 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.929515    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" path="/var/lib/kubelet/pods/3653f1be-c7ce-41bf-a5c8-1fdd40762aeb/volumes"
Nov 03 18:15:02.930130 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.930110    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" path="/var/lib/kubelet/pods/3cd797ae-f45d-4466-ab72-fe20b4af183c/volumes"
Nov 03 18:15:02.954043 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:02.954010    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"]
Nov 03 18:15:03.004157 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:03.004122    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-8zgzs"]
Nov 03 18:15:04.929438 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:04.929403    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" path="/var/lib/kubelet/pods/21faf221-c045-4d19-88ca-6a2fd72667be/volumes"
Nov 03 18:15:07.571619 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.571585    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"]
Nov 03 18:15:07.572370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572354    6167 cpu_manager.go:414] "containerMap: removing container" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy"
Nov 03 18:15:07.572427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572371    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy"
Nov 03 18:15:07.572427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572383    6167 cpu_manager.go:414] "containerMap: removing container" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy"
Nov 03 18:15:07.572427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572392    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy"
Nov 03 18:15:07.572427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572417    6167 cpu_manager.go:414] "containerMap: removing container" podUID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" containerName="maas-api"
Nov 03 18:15:07.572427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572423    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" containerName="maas-api"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572434    6167 cpu_manager.go:414] "containerMap: removing container" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572439    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572460    6167 cpu_manager.go:414] "containerMap: removing container" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572464    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572581    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb280431-6efb-4ee4-8446-ef91773809b3" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572590    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572598    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572607    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca50e99-73fa-4476-9fe3-abcdcc4b92a1" containerName="maas-api"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572613    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd797ae-f45d-4466-ab72-fe20b4af183c" containerName="istio-proxy"
Nov 03 18:15:07.572664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.572622    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="3653f1be-c7ce-41bf-a5c8-1fdd40762aeb" containerName="istio-proxy"
Nov 03 18:15:07.576440 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.576417    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.632777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.632608    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"]
Nov 03 18:15:07.654624 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.654589    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.654813 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.654772    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.654922 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.654821    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78gb5\" (UniqueName: \"kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.654982 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.654924    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.677956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.677930    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-232f9e-serving-cert\""
Nov 03 18:15:07.756438 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.756388    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.756627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.756455    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gb5\" (UniqueName: \"kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.756627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.756499    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.756627 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:07.756540    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-serving-cert: secret "maas-api-e2e-shared-b-w5-232f9e-serving-cert" not found
Nov 03 18:15:07.756627 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:07.756612    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls podName:11f7da85-fbe3-47ec-8bb5-4e138ac94e86 nodeName:}" failed. No retries permitted until 2026-11-03 18:15:08.256592322 +0000 UTC m=+1763.486917377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls") pod "maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" (UID: "11f7da85-fbe3-47ec-8bb5-4e138ac94e86") : secret "maas-api-e2e-shared-b-w5-232f9e-serving-cert" not found
Nov 03 18:15:07.756941 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.756775    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.757633 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.757602    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.759504 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.759458    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:07.819190 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:07.819158    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gb5\" (UniqueName: \"kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:08.280131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:08.280091    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:08.282569 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:08.282547    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:08.491554 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:08.491513    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:08.871224 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:15:08.871183    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f7da85_fbe3_47ec_8bb5_4e138ac94e86.slice/crio-bbb4e51d98ef8cb06223e1ee6d84b8abbe815ff2b974624c216dce0de4440b8b WatchSource:0}: Error finding container bbb4e51d98ef8cb06223e1ee6d84b8abbe815ff2b974624c216dce0de4440b8b: Status 404 returned error can't find the container with id bbb4e51d98ef8cb06223e1ee6d84b8abbe815ff2b974624c216dce0de4440b8b
Nov 03 18:15:08.876538 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:08.876506    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"]
Nov 03 18:15:09.844789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:09.844742    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" event={"ID":"11f7da85-fbe3-47ec-8bb5-4e138ac94e86","Type":"ContainerStarted","Data":"5cb0ad5b465af6258e2a5b4ef397d2574c0bca0caf34173a7f251acfd9ef83c3"}
Nov 03 18:15:09.844957 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:09.844815    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:09.844957 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:09.844838    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" event={"ID":"11f7da85-fbe3-47ec-8bb5-4e138ac94e86","Type":"ContainerStarted","Data":"bbb4e51d98ef8cb06223e1ee6d84b8abbe815ff2b974624c216dce0de4440b8b"}
Nov 03 18:15:09.982435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:09.982336    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" podStartSLOduration=2.162994895 podStartE2EDuration="2.982322035s" totalImagesPullingTime="819.32714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:09.979173237 +0000 UTC m=+1765.209498300" watchObservedRunningTime="2026-11-03 18:15:09.982322035 +0000 UTC m=+1765.212647138"
Nov 03 18:15:12.284458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.284415    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"]
Nov 03 18:15:12.285118 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.284740    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" podUID="89064973-1a13-4dca-80b5-6e9ee9c1d9fa" containerName="maas-api" containerID="cri-o://f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81" gracePeriod=30
Nov 03 18:15:12.599562 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.599534    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 18:15:12.655869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.655842    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q\" (UniqueName: \"kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q\") pod \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") "
Nov 03 18:15:12.656013 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.655909    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs\") pod \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") "
Nov 03 18:15:12.656075 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.656059    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca\") pod \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") "
Nov 03 18:15:12.656152 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.656137    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\") pod \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\" (UID: \"89064973-1a13-4dca-80b5-6e9ee9c1d9fa\") "
Nov 03 18:15:12.656421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.656397    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca" pod "89064973-1a13-4dca-80b5-6e9ee9c1d9fa" (UID: "89064973-1a13-4dca-80b5-6e9ee9c1d9fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:12.656691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.656673    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:12.658091 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.658065    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs" pod "89064973-1a13-4dca-80b5-6e9ee9c1d9fa" (UID: "89064973-1a13-4dca-80b5-6e9ee9c1d9fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:12.658172 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.658110    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q" pod "89064973-1a13-4dca-80b5-6e9ee9c1d9fa" (UID: "89064973-1a13-4dca-80b5-6e9ee9c1d9fa"). InnerVolumeSpecName "kube-api-access-6cw5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:12.658172 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.658118    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls" pod "89064973-1a13-4dca-80b5-6e9ee9c1d9fa" (UID: "89064973-1a13-4dca-80b5-6e9ee9c1d9fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:12.758730 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.758687    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cw5q\" (UniqueName: \"kubernetes.io/projected/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-kube-api-access-6cw5q\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:12.758889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.758743    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:12.758889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.758759    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89064973-1a13-4dca-80b5-6e9ee9c1d9fa-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:12.873088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.873045    6167 generic.go:356] "Generic (PLEG): container finished" podID="89064973-1a13-4dca-80b5-6e9ee9c1d9fa" containerID="f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81" exitCode=0
Nov 03 18:15:12.873951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.873150    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"
Nov 03 18:15:12.873951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.873215    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" event={"ID":"89064973-1a13-4dca-80b5-6e9ee9c1d9fa","Type":"ContainerDied","Data":"f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81"}
Nov 03 18:15:12.873951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.873353    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv" event={"ID":"89064973-1a13-4dca-80b5-6e9ee9c1d9fa","Type":"ContainerDied","Data":"656525aa77f961356bbf6f2181d9de88de9349efce56167cccb8144ac1a275f3"}
Nov 03 18:15:12.873951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.873421    6167 scope.go:122] "RemoveContainer" containerID="f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81"
Nov 03 18:15:12.887823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.887786    6167 scope.go:122] "RemoveContainer" containerID="f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81"
Nov 03 18:15:12.888231 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:12.888195    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81\": container with ID starting with f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81 not found: ID does not exist" containerID="f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81"
Nov 03 18:15:12.888330 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:12.888239    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81"} err="failed to get container status \"f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81\": rpc error: code = NotFound desc = could not find container \"f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81\": container with ID starting with f7f7742c0f43e03c0e314bf033f47addd9b50c91f8e57bf49fbb2b5688c8aa81 not found: ID does not exist"
Nov 03 18:15:13.045524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:13.045488    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"]
Nov 03 18:15:13.053516 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:13.052880    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0adcbc-6dd7cbcbff-6mggv"]
Nov 03 18:15:14.805143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:14.805100    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"]
Nov 03 18:15:14.936156 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:14.936126    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89064973-1a13-4dca-80b5-6e9ee9c1d9fa" path="/var/lib/kubelet/pods/89064973-1a13-4dca-80b5-6e9ee9c1d9fa/volumes"
Nov 03 18:15:15.857473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:15.857445    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:15:17.874415 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:17.874381    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"]
Nov 03 18:15:17.874895 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:17.874590    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy" containerID="cri-o://ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a" gracePeriod=30
Nov 03 18:15:19.806550 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:19.806494    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerName="payload-pre-processing" containerID="cri-o://a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5" gracePeriod=25
Nov 03 18:15:22.116923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.116885    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"]
Nov 03 18:15:22.117361 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.117177    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" containerID="cri-o://8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2" gracePeriod=30
Nov 03 18:15:22.141884 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.141851    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"]
Nov 03 18:15:22.142303 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.142267    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" containerID="cri-o://0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73" gracePeriod=30
Nov 03 18:15:22.168218 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.168187    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"]
Nov 03 18:15:22.168461 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.168425    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy" containerID="cri-o://866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf" gracePeriod=30
Nov 03 18:15:22.801019 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.800983    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:15:22.801204 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.801040    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:22.807955 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.807924    6167 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:15:22.808046 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:22.807995    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:23.183158 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.183128    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:15:23.341762 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.341706    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.341965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.341831    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.341965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.341875    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.341965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.341906    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.342183 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342060    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.342183 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342116    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.342310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342188    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.342310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342215    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.342310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342295    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m\") pod \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\" (UID: \"f40f64c2-53b5-4d1d-9f9d-b9207429f502\") "
Nov 03 18:15:23.342463 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342403    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:23.342770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.342746    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:23.343094 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.343066    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:23.343176 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.343127    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.343176 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.343147    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.343300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.343274    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:23.343368 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.343345    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:23.345326 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.345288    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "kube-api-access-fch9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:23.345602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.345575    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:23.345890 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.345867    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:23.347558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.347531    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token" pod "f40f64c2-53b5-4d1d-9f9d-b9207429f502" (UID: "f40f64c2-53b5-4d1d-9f9d-b9207429f502"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:23.444770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444704    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fch9m\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-kube-api-access-fch9m\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.444770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444756    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.444770 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444771    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.445073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444784    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.445073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444794    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.445073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444804    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.445073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.444811    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f64c2-53b5-4d1d-9f9d-b9207429f502-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:23.998297 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.992947    6167 generic.go:356] "Generic (PLEG): container finished" podID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerID="ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a" exitCode=0
Nov 03 18:15:23.998297 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.993177    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" event={"ID":"f40f64c2-53b5-4d1d-9f9d-b9207429f502","Type":"ContainerDied","Data":"ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a"}
Nov 03 18:15:23.998297 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.993230    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5" event={"ID":"f40f64c2-53b5-4d1d-9f9d-b9207429f502","Type":"ContainerDied","Data":"3c409ddb5b2e4cfa0e12a22301788acd4fb810eff0c8f68d7929e8a0488fb3ab"}
Nov 03 18:15:23.998297 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.993260    6167 scope.go:122] "RemoveContainer" containerID="ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a"
Nov 03 18:15:23.998297 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:23.993433    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"
Nov 03 18:15:24.029156 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:24.029104    6167 scope.go:122] "RemoveContainer" containerID="ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a"
Nov 03 18:15:24.029545 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:24.029520    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a\": container with ID starting with ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a not found: ID does not exist" containerID="ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a"
Nov 03 18:15:24.029659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:24.029554    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a"} err="failed to get container status \"ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a\": rpc error: code = NotFound desc = could not find container \"ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a\": container with ID starting with ed7eed0124231fbb0fff3b76c336bb8047c16834b39be5ef10657faa9b3d6f1a not found: ID does not exist"
Nov 03 18:15:24.168434 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:24.168358    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"]
Nov 03 18:15:24.223850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:24.223785    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-b5cr5"]
Nov 03 18:15:24.421865 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:24.421819    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:24.527977 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:24.527942    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:15:24.929267 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:24.929230    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" path="/var/lib/kubelet/pods/f40f64c2-53b5-4d1d-9f9d-b9207429f502/volumes"
Nov 03 18:15:27.482492 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.482080    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:15:27.486647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.486613    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:15:27.530963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.530931    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.531142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531002    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.531142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531066    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.531142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531122    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.531364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531165    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.531364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531198    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.531364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531246    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.531364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531293    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.531616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531393    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.531616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531548    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.531616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531584    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.531793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531638    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.531793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531669    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.531793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531755    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.531793 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531784    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.532017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531788    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:27.532017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531828    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.532017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531871    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:27.532017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531880    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.532017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531961    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket\") pod \"5501f821-1679-48b9-9bde-4dbaaf0450fb\" (UID: \"5501f821-1679-48b9-9bde-4dbaaf0450fb\") "
Nov 03 18:15:27.532017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.531990    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.532335 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.532051    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs\") pod \"d5272092-0dea-4ffd-93e1-e64256cffbd5\" (UID: \"d5272092-0dea-4ffd-93e1-e64256cffbd5\") "
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.532565    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.532807    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.532989    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533393    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5501f821-1679-48b9-9bde-4dbaaf0450fb-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533420    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533433    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533447    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533459    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533473    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5272092-0dea-4ffd-93e1-e64256cffbd5-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533694    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.534042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.533947    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "kube-api-access-w5h7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.534576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.534086    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.534576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.534199    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.534576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.534565    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.534768 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.534641    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.534903 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.534876    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:27.535115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.535095    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.535296 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.535276    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.535870 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.535845    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh" pod "5501f821-1679-48b9-9bde-4dbaaf0450fb" (UID: "5501f821-1679-48b9-9bde-4dbaaf0450fb"). InnerVolumeSpecName "kube-api-access-nhgxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.536265 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.536243    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo" pod "d5272092-0dea-4ffd-93e1-e64256cffbd5" (UID: "d5272092-0dea-4ffd-93e1-e64256cffbd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:27.551316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.551291    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:15:27.633884 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.633852    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.633924    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634027    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634080    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634138    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634196    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634257    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.634378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634276    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634296    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:27.634378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634309    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634533 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634376    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs\") pod \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\" (UID: \"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1\") "
Nov 03 18:15:27.634533 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634465    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.635008 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.634969    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.635137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635055    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635079    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635096    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5h7k\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-kube-api-access-w5h7k\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635110    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635123    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635137    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635151    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635164    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635179    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635185    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635191    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635231    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635253    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhgxh\" (UniqueName: \"kubernetes.io/projected/5501f821-1679-48b9-9bde-4dbaaf0450fb-kube-api-access-nhgxh\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635269    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635285    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5501f821-1679-48b9-9bde-4dbaaf0450fb-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.635462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.635299    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5272092-0dea-4ffd-93e1-e64256cffbd5-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.637091 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.637065    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:27.637233 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.637119    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.637307 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.637287    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:27.637453 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.637423    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz" pod "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" (UID: "ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1"). InnerVolumeSpecName "kube-api-access-s9xcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:27.736313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.736219    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.736313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.736253    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.736313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.736266    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.736313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.736278    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9xcz\" (UniqueName: \"kubernetes.io/projected/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-kube-api-access-s9xcz\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.736313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.736292    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:27.736313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:27.736304    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:28.028229 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.028147    6167 generic.go:356] "Generic (PLEG): container finished" podID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerID="8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2" exitCode=0
Nov 03 18:15:28.028229 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.028196    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"
Nov 03 18:15:28.028408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.028229    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" event={"ID":"d5272092-0dea-4ffd-93e1-e64256cffbd5","Type":"ContainerDied","Data":"8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2"}
Nov 03 18:15:28.028408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.028316    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7" event={"ID":"d5272092-0dea-4ffd-93e1-e64256cffbd5","Type":"ContainerDied","Data":"21207f6a226f2452c9ef84e961142e551309a260754036a8d4ff8e504bdde392"}
Nov 03 18:15:28.028408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.028333    6167 scope.go:122] "RemoveContainer" containerID="8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2"
Nov 03 18:15:28.029794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.029770    6167 generic.go:356] "Generic (PLEG): container finished" podID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerID="866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf" exitCode=0
Nov 03 18:15:28.029913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.029814    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"
Nov 03 18:15:28.029913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.029872    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" event={"ID":"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1","Type":"ContainerDied","Data":"866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf"}
Nov 03 18:15:28.030038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.029926    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk" event={"ID":"ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1","Type":"ContainerDied","Data":"4364fcf1301a0140dd3a3e5c988160f971f5a7e8220f7fe3f78080607d53f145"}
Nov 03 18:15:28.031600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.031575    6167 generic.go:356] "Generic (PLEG): container finished" podID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerID="0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73" exitCode=0
Nov 03 18:15:28.031745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.031608    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" event={"ID":"5501f821-1679-48b9-9bde-4dbaaf0450fb","Type":"ContainerDied","Data":"0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73"}
Nov 03 18:15:28.031745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.031637    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"
Nov 03 18:15:28.031745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.031660    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl" event={"ID":"5501f821-1679-48b9-9bde-4dbaaf0450fb","Type":"ContainerDied","Data":"473b6aa2183c9a9b8f248abfc6518ad4e9e14a8e8f8e4f094318568f35b082c7"}
Nov 03 18:15:28.039889 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.039868    6167 scope.go:122] "RemoveContainer" containerID="8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2"
Nov 03 18:15:28.040192 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:28.040162    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2\": container with ID starting with 8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2 not found: ID does not exist" containerID="8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2"
Nov 03 18:15:28.040276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.040189    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2"} err="failed to get container status \"8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2\": rpc error: code = NotFound desc = could not find container \"8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2\": container with ID starting with 8fb5c60051a7a5384ee0e636a61237c10941825d9ccaca47e39865d7d84af1f2 not found: ID does not exist"
Nov 03 18:15:28.040276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.040206    6167 scope.go:122] "RemoveContainer" containerID="866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf"
Nov 03 18:15:28.049750 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.049711    6167 scope.go:122] "RemoveContainer" containerID="866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf"
Nov 03 18:15:28.049998 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:28.049978    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf\": container with ID starting with 866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf not found: ID does not exist" containerID="866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf"
Nov 03 18:15:28.050076 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.050007    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf"} err="failed to get container status \"866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf\": rpc error: code = NotFound desc = could not find container \"866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf\": container with ID starting with 866e7895cd4fe563614d36df1125ecb5f107d311f7efc1b2626468b48ad93fcf not found: ID does not exist"
Nov 03 18:15:28.050076 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.050029    6167 scope.go:122] "RemoveContainer" containerID="0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73"
Nov 03 18:15:28.058847 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.058829    6167 scope.go:122] "RemoveContainer" containerID="0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73"
Nov 03 18:15:28.059095 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:28.059073    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73\": container with ID starting with 0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73 not found: ID does not exist" containerID="0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73"
Nov 03 18:15:28.059171 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.059099    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73"} err="failed to get container status \"0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73\": rpc error: code = NotFound desc = could not find container \"0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73\": container with ID starting with 0698eeab279a72476dc20ae737f514c40a91dc97e622c876d65a93a7da47fd73 not found: ID does not exist"
Nov 03 18:15:28.199002 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.198967    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"]
Nov 03 18:15:28.202600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.202575    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-lr4bk"]
Nov 03 18:15:28.312898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.312824    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"]
Nov 03 18:15:28.321772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.321749    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-9xhq7"]
Nov 03 18:15:28.434269 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.434237    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"]
Nov 03 18:15:28.444431 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.444404    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-vfzrl"]
Nov 03 18:15:28.930296 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.930264    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" path="/var/lib/kubelet/pods/5501f821-1679-48b9-9bde-4dbaaf0450fb/volumes"
Nov 03 18:15:28.930767 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.930753    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" path="/var/lib/kubelet/pods/ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1/volumes"
Nov 03 18:15:28.931184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:28.931172    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" path="/var/lib/kubelet/pods/d5272092-0dea-4ffd-93e1-e64256cffbd5/volumes"
Nov 03 18:15:34.421802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:34.421765    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:40.093495 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.093469    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 18:15:40.122045 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.122011    6167 generic.go:356] "Generic (PLEG): container finished" podID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerID="a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5" exitCode=0
Nov 03 18:15:40.122197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.122061    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"
Nov 03 18:15:40.122197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.122087    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" event={"ID":"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a","Type":"ContainerDied","Data":"a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5"}
Nov 03 18:15:40.122197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.122154    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674" event={"ID":"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a","Type":"ContainerDied","Data":"8b24276c507c0f06ecaa97f9a892adb262ffca748c24cf29f300d9643e4e7c23"}
Nov 03 18:15:40.122197 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.122179    6167 scope.go:122] "RemoveContainer" containerID="a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5"
Nov 03 18:15:40.137568 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.137541    6167 scope.go:122] "RemoveContainer" containerID="a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5"
Nov 03 18:15:40.138043 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:40.138006    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5\": container with ID starting with a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5 not found: ID does not exist" containerID="a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5"
Nov 03 18:15:40.138174 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.138066    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5"} err="failed to get container status \"a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5\": rpc error: code = NotFound desc = could not find container \"a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5\": container with ID starting with a58d15cf02c1d370c9379e56e9a89cbc9d7c2d597e64b517cf66a6a1baf53fd5 not found: ID does not exist"
Nov 03 18:15:40.202658 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.202622    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume\") pod \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") "
Nov 03 18:15:40.202856 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.202847    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m\" (UniqueName: \"kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m\") pod \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\" (UID: \"2ad2dad4-78e8-4a6f-81f9-1c004379ee3a\") "
Nov 03 18:15:40.203039 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.203018    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume" pod "2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" (UID: "2ad2dad4-78e8-4a6f-81f9-1c004379ee3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:40.203313 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.203298    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:40.204899 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.204879    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m" pod "2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" (UID: "2ad2dad4-78e8-4a6f-81f9-1c004379ee3a"). InnerVolumeSpecName "kube-api-access-twn8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:40.304595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.304558    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twn8m\" (UniqueName: \"kubernetes.io/projected/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a-kube-api-access-twn8m\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:40.553616 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.553575    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"]
Nov 03 18:15:40.557517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.557485    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a86228-77f5cf9fc9-sf674"]
Nov 03 18:15:40.928485 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:40.928443    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" path="/var/lib/kubelet/pods/2ad2dad4-78e8-4a6f-81f9-1c004379ee3a/volumes"
Nov 03 18:15:45.306961 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:45.306930    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:15:45.312095 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:45.312065    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:15:47.595610 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:47.595574    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"]
Nov 03 18:15:51.212814 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.211469    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"]
Nov 03 18:15:51.212814 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.211696    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" podUID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" containerName="maas-api" containerID="cri-o://e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b" gracePeriod=30
Nov 03 18:15:51.325244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.313808    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" podUID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.64:8443/health\": dial tcp 10.132.0.64:8443: connect: connection refused"
Nov 03 18:15:51.519427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.519398    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 18:15:51.583531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.583500    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp\" (UniqueName: \"kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp\") pod \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") "
Nov 03 18:15:51.583792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.583567    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs\") pod \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") "
Nov 03 18:15:51.583792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.583653    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca\") pod \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") "
Nov 03 18:15:51.583928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.583839    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\") pod \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\" (UID: \"207b02f4-b43d-4c8a-9dd8-fd77189406dc\") "
Nov 03 18:15:51.584120 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.584093    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca" pod "207b02f4-b43d-4c8a-9dd8-fd77189406dc" (UID: "207b02f4-b43d-4c8a-9dd8-fd77189406dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:51.584399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.584373    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/207b02f4-b43d-4c8a-9dd8-fd77189406dc-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:51.585921 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.585898    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls" pod "207b02f4-b43d-4c8a-9dd8-fd77189406dc" (UID: "207b02f4-b43d-4c8a-9dd8-fd77189406dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:51.586094 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.586066    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp" pod "207b02f4-b43d-4c8a-9dd8-fd77189406dc" (UID: "207b02f4-b43d-4c8a-9dd8-fd77189406dc"). InnerVolumeSpecName "kube-api-access-s5tjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:51.586180 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.586144    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs" pod "207b02f4-b43d-4c8a-9dd8-fd77189406dc" (UID: "207b02f4-b43d-4c8a-9dd8-fd77189406dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:51.685706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.685669    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:51.685706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.685697    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5tjp\" (UniqueName: \"kubernetes.io/projected/207b02f4-b43d-4c8a-9dd8-fd77189406dc-kube-api-access-s5tjp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:51.685706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:51.685708    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/207b02f4-b43d-4c8a-9dd8-fd77189406dc-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.215247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.215209    6167 generic.go:356] "Generic (PLEG): container finished" podID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" containerID="e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b" exitCode=0
Nov 03 18:15:52.215802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.215263    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"
Nov 03 18:15:52.215802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.215286    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" event={"ID":"207b02f4-b43d-4c8a-9dd8-fd77189406dc","Type":"ContainerDied","Data":"e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b"}
Nov 03 18:15:52.215802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.215361    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s" event={"ID":"207b02f4-b43d-4c8a-9dd8-fd77189406dc","Type":"ContainerDied","Data":"525ed2588d636152493089a0a31edd9439e34b8a4550782b09fba92a0df73755"}
Nov 03 18:15:52.215802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.215396    6167 scope.go:122] "RemoveContainer" containerID="e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b"
Nov 03 18:15:52.226622 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.226603    6167 scope.go:122] "RemoveContainer" containerID="e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b"
Nov 03 18:15:52.226913 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:52.226893    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b\": container with ID starting with e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b not found: ID does not exist" containerID="e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b"
Nov 03 18:15:52.226995 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.226921    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b"} err="failed to get container status \"e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b\": rpc error: code = NotFound desc = could not find container \"e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b\": container with ID starting with e6839e325999969caa05603fccc17e6107ac4796cbdb9757735a104659a65a9b not found: ID does not exist"
Nov 03 18:15:52.288350 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.288312    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"]
Nov 03 18:15:52.288594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.288549    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" podUID="b2db6fdf-7669-4ed8-9299-4100c1850311" containerName="maas-api" containerID="cri-o://af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996" gracePeriod=30
Nov 03 18:15:52.383645 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.383610    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"]
Nov 03 18:15:52.386395 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.386369    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-aab753-b965c95b6-k4s2s"]
Nov 03 18:15:52.596523 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.596490    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5" podUID="a912bc62-c39d-4a84-8092-055d55a2f59f" containerName="payload-processing" containerID="cri-o://dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b" gracePeriod=25
Nov 03 18:15:52.662022 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.661992    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 18:15:52.698224 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.698193    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\") pod \"b2db6fdf-7669-4ed8-9299-4100c1850311\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") "
Nov 03 18:15:52.698386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.698232    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk\" (UniqueName: \"kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk\") pod \"b2db6fdf-7669-4ed8-9299-4100c1850311\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") "
Nov 03 18:15:52.698386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.698321    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs\") pod \"b2db6fdf-7669-4ed8-9299-4100c1850311\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") "
Nov 03 18:15:52.698509 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.698454    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca\") pod \"b2db6fdf-7669-4ed8-9299-4100c1850311\" (UID: \"b2db6fdf-7669-4ed8-9299-4100c1850311\") "
Nov 03 18:15:52.698826 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.698794    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca" pod "b2db6fdf-7669-4ed8-9299-4100c1850311" (UID: "b2db6fdf-7669-4ed8-9299-4100c1850311"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:52.699027 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.699008    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2db6fdf-7669-4ed8-9299-4100c1850311-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.700489 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.700383    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls" pod "b2db6fdf-7669-4ed8-9299-4100c1850311" (UID: "b2db6fdf-7669-4ed8-9299-4100c1850311"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:52.700489 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.700463    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs" pod "b2db6fdf-7669-4ed8-9299-4100c1850311" (UID: "b2db6fdf-7669-4ed8-9299-4100c1850311"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:52.700800 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.700771    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk" pod "b2db6fdf-7669-4ed8-9299-4100c1850311" (UID: "b2db6fdf-7669-4ed8-9299-4100c1850311"). InnerVolumeSpecName "kube-api-access-8jztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:52.803875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.799819    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.803875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.799847    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jztk\" (UniqueName: \"kubernetes.io/projected/b2db6fdf-7669-4ed8-9299-4100c1850311-kube-api-access-8jztk\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.803875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.799857    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2db6fdf-7669-4ed8-9299-4100c1850311-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.901049 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.901022    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:15:52.928501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:52.928473    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" path="/var/lib/kubelet/pods/207b02f4-b43d-4c8a-9dd8-fd77189406dc/volumes"
Nov 03 18:15:53.001264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.001231    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr\" (UniqueName: \"kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr\") pod \"a912bc62-c39d-4a84-8092-055d55a2f59f\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") "
Nov 03 18:15:53.001429 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.001329    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume\") pod \"a912bc62-c39d-4a84-8092-055d55a2f59f\" (UID: \"a912bc62-c39d-4a84-8092-055d55a2f59f\") "
Nov 03 18:15:53.001788 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.001759    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume" pod "a912bc62-c39d-4a84-8092-055d55a2f59f" (UID: "a912bc62-c39d-4a84-8092-055d55a2f59f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:53.003435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.003410    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr" pod "a912bc62-c39d-4a84-8092-055d55a2f59f" (UID: "a912bc62-c39d-4a84-8092-055d55a2f59f"). InnerVolumeSpecName "kube-api-access-4dcbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:53.103521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.103486    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcbr\" (UniqueName: \"kubernetes.io/projected/a912bc62-c39d-4a84-8092-055d55a2f59f-kube-api-access-4dcbr\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.103521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.103517    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a912bc62-c39d-4a84-8092-055d55a2f59f-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.270786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.265692    6167 generic.go:356] "Generic (PLEG): container finished" podID="a912bc62-c39d-4a84-8092-055d55a2f59f" containerID="dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b" exitCode=0
Nov 03 18:15:53.270786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.265853    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5" event={"ID":"a912bc62-c39d-4a84-8092-055d55a2f59f","Type":"ContainerDied","Data":"dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b"}
Nov 03 18:15:53.270786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.265905    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5" event={"ID":"a912bc62-c39d-4a84-8092-055d55a2f59f","Type":"ContainerDied","Data":"6906be491f383d4e197ef80ae105c123ac443e4b18768fba9bede991bbddaf11"}
Nov 03 18:15:53.270786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.265933    6167 scope.go:122] "RemoveContainer" containerID="dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b"
Nov 03 18:15:53.270786 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.266032    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"
Nov 03 18:15:53.285886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.272314    6167 generic.go:356] "Generic (PLEG): container finished" podID="b2db6fdf-7669-4ed8-9299-4100c1850311" containerID="af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996" exitCode=0
Nov 03 18:15:53.285886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.272438    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" event={"ID":"b2db6fdf-7669-4ed8-9299-4100c1850311","Type":"ContainerDied","Data":"af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996"}
Nov 03 18:15:53.285886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.272485    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz" event={"ID":"b2db6fdf-7669-4ed8-9299-4100c1850311","Type":"ContainerDied","Data":"2750d0ab129bc4dabc47bda3ca34da88b514b4d5866f5fb6a902d30ec2a288f5"}
Nov 03 18:15:53.285886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.272558    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"
Nov 03 18:15:53.309186 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.309161    6167 scope.go:122] "RemoveContainer" containerID="dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b"
Nov 03 18:15:53.309636 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:53.309602    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b\": container with ID starting with dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b not found: ID does not exist" containerID="dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b"
Nov 03 18:15:53.309906 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.309640    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b"} err="failed to get container status \"dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b\": rpc error: code = NotFound desc = could not find container \"dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b\": container with ID starting with dfe749b7f05d21e63c297a872a8b764b95be117ff31d5d3c5a39230730ce867b not found: ID does not exist"
Nov 03 18:15:53.309906 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.309663    6167 scope.go:122] "RemoveContainer" containerID="af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996"
Nov 03 18:15:53.323477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.323454    6167 scope.go:122] "RemoveContainer" containerID="af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996"
Nov 03 18:15:53.323817 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:53.323786    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996\": container with ID starting with af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996 not found: ID does not exist" containerID="af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996"
Nov 03 18:15:53.323915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.323824    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996"} err="failed to get container status \"af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996\": rpc error: code = NotFound desc = could not find container \"af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996\": container with ID starting with af9a7eb50e3825c13450366364be1560c9eda84d0691cde4fb4fc54379226996 not found: ID does not exist"
Nov 03 18:15:53.399260 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.399226    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"]
Nov 03 18:15:53.399469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.399448    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" podUID="a1a6f824-e9f5-4f47-a8f2-74b4e266c174" containerName="maas-api" containerID="cri-o://ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f" gracePeriod=30
Nov 03 18:15:53.501386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.501355    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"]
Nov 03 18:15:53.506049 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.506023    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a86228-6f59cfd6db-v9ctz"]
Nov 03 18:15:53.625677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.623820    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"]
Nov 03 18:15:53.641543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.641500    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7a7c76-6c778969cf-dbpp5"]
Nov 03 18:15:53.708552 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.708525    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:15:53.822768 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.822705    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49\" (UniqueName: \"kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49\") pod \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") "
Nov 03 18:15:53.822939 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.822789    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca\") pod \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") "
Nov 03 18:15:53.822939 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.822880    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs\") pod \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") "
Nov 03 18:15:53.823055 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.822973    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\") pod \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\" (UID: \"a1a6f824-e9f5-4f47-a8f2-74b4e266c174\") "
Nov 03 18:15:53.823187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.823159    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca" pod "a1a6f824-e9f5-4f47-a8f2-74b4e266c174" (UID: "a1a6f824-e9f5-4f47-a8f2-74b4e266c174"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:53.823689 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.823617    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.825094 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.825067    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls" pod "a1a6f824-e9f5-4f47-a8f2-74b4e266c174" (UID: "a1a6f824-e9f5-4f47-a8f2-74b4e266c174"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:53.825178 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.825133    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49" pod "a1a6f824-e9f5-4f47-a8f2-74b4e266c174" (UID: "a1a6f824-e9f5-4f47-a8f2-74b4e266c174"). InnerVolumeSpecName "kube-api-access-cbm49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:53.825339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.825316    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs" pod "a1a6f824-e9f5-4f47-a8f2-74b4e266c174" (UID: "a1a6f824-e9f5-4f47-a8f2-74b4e266c174"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:53.924858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.924823    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbm49\" (UniqueName: \"kubernetes.io/projected/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-kube-api-access-cbm49\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.924858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.924856    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.925060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:53.924870    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a6f824-e9f5-4f47-a8f2-74b4e266c174-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:15:54.288068 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.287973    6167 generic.go:356] "Generic (PLEG): container finished" podID="a1a6f824-e9f5-4f47-a8f2-74b4e266c174" containerID="ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f" exitCode=0
Nov 03 18:15:54.288556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.288098    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" event={"ID":"a1a6f824-e9f5-4f47-a8f2-74b4e266c174","Type":"ContainerDied","Data":"ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f"}
Nov 03 18:15:54.288556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.288148    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n" event={"ID":"a1a6f824-e9f5-4f47-a8f2-74b4e266c174","Type":"ContainerDied","Data":"34c37157f3362fb1a03498eb7524951acb020c07721e2fdc035f123305f67a0a"}
Nov 03 18:15:54.288556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.288178    6167 scope.go:122] "RemoveContainer" containerID="ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f"
Nov 03 18:15:54.288556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.288329    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"
Nov 03 18:15:54.363606 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.363576    6167 scope.go:122] "RemoveContainer" containerID="ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f"
Nov 03 18:15:54.364241 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:54.364214    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f\": container with ID starting with ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f not found: ID does not exist" containerID="ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f"
Nov 03 18:15:54.364415 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.364386    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f"} err="failed to get container status \"ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f\": rpc error: code = NotFound desc = could not find container \"ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f\": container with ID starting with ef69ef102e10d6559f457d684f9e38dd968da06aa42d67c28bd65212f36ee35f not found: ID does not exist"
Nov 03 18:15:54.455912 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.455866    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"]
Nov 03 18:15:54.461402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.461373    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7a7c76-bd755d56f-vf47n"]
Nov 03 18:15:54.522523 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:15:54.522485    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:15:54.929249 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.929219    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a6f824-e9f5-4f47-a8f2-74b4e266c174" path="/var/lib/kubelet/pods/a1a6f824-e9f5-4f47-a8f2-74b4e266c174/volumes"
Nov 03 18:15:54.929624 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.929610    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a912bc62-c39d-4a84-8092-055d55a2f59f" path="/var/lib/kubelet/pods/a912bc62-c39d-4a84-8092-055d55a2f59f/volumes"
Nov 03 18:15:54.930091 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:15:54.930073    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2db6fdf-7669-4ed8-9299-4100c1850311" path="/var/lib/kubelet/pods/b2db6fdf-7669-4ed8-9299-4100c1850311/volumes"
Nov 03 18:16:01.124477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:01.124437    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"]
Nov 03 18:16:01.125074 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:01.124823    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" containerID="cri-o://4a67a8e20f39b7f93dd251b412ebcd1f6ca98c21ec69f23aae06d091754c1b1b" gracePeriod=30
Nov 03 18:16:01.137492 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:01.137465    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"]
Nov 03 18:16:01.137665 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:01.137646    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy" containerID="cri-o://33947b4d0cf5fe421be106e8693eb4385c6675b24aa4fa839ce6697d60af9e20" gracePeriod=30
Nov 03 18:16:02.197119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:02.193092    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"]
Nov 03 18:16:02.197119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:02.193334    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" containerID="cri-o://a21346ba92ab4667edbe008dacbf8934aaaa5fe23de8565c14b2ff12ebd9e03f" gracePeriod=30
Nov 03 18:16:02.198578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:02.197651    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"]
Nov 03 18:16:02.198578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:02.197992    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" containerID="cri-o://17a0efcbfb837a885d8c001ad03634508a3d74b7202852a87f9d01ebf505f215" gracePeriod=30
Nov 03 18:16:02.225046 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:02.225008    6167 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:16:02.225230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:02.225084    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:16:05.368328 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:05.368282    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:16:05.368757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:05.368347    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:16:06.322992 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.322683    6167 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:16:06.322992 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.322765    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:16:06.376515 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.376268    6167 generic.go:356] "Generic (PLEG): container finished" podID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerID="33947b4d0cf5fe421be106e8693eb4385c6675b24aa4fa839ce6697d60af9e20" exitCode=0
Nov 03 18:16:06.376515 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.376439    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" event={"ID":"6e61eebf-fb19-4f5f-80ae-9bd904347052","Type":"ContainerDied","Data":"33947b4d0cf5fe421be106e8693eb4385c6675b24aa4fa839ce6697d60af9e20"}
Nov 03 18:16:06.379850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.379819    6167 generic.go:356] "Generic (PLEG): container finished" podID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerID="4a67a8e20f39b7f93dd251b412ebcd1f6ca98c21ec69f23aae06d091754c1b1b" exitCode=0
Nov 03 18:16:06.379994 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.379967    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" event={"ID":"38d3bbfc-f466-49d3-b876-910aa65efd6a","Type":"ContainerDied","Data":"4a67a8e20f39b7f93dd251b412ebcd1f6ca98c21ec69f23aae06d091754c1b1b"}
Nov 03 18:16:06.380058 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.380017    6167 scope.go:122] "RemoveContainer" containerID="863a366185789d3e8b6a4030e0c28334a7240f10ce8c4509be8297c524891c1d"
Nov 03 18:16:06.438811 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.438762    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 18:16:06.458686 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.458659    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:16:06.524776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.524742    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.524945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.524797    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.524945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.524878    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.524945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.524933    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.524963    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525006    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525106    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525115    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.525355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525145    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525223    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.525355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525239    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525292    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525342    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525422    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525453    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.525629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525494    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525567    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525604    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525756    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525790    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs\") pod \"6e61eebf-fb19-4f5f-80ae-9bd904347052\" (UID: \"6e61eebf-fb19-4f5f-80ae-9bd904347052\") "
Nov 03 18:16:06.525915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525842    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token\") pod \"38d3bbfc-f466-49d3-b876-910aa65efd6a\" (UID: \"38d3bbfc-f466-49d3-b876-910aa65efd6a\") "
Nov 03 18:16:06.525915 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.525889    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.526122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526036    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526296    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526388    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526224    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526691    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526711    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526743    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526757    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526771    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526786    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e61eebf-fb19-4f5f-80ae-9bd904347052-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.526823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526801    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.527435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.526848    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.527435 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.527030    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.529316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.529286    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:06.529572 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.529548    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.529659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.529630    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:06.529794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.529766    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "kube-api-access-c4ck6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.530142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.530121    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.539954 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.539909    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy" pod "6e61eebf-fb19-4f5f-80ae-9bd904347052" (UID: "6e61eebf-fb19-4f5f-80ae-9bd904347052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.540049 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.539978    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:06.540049 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.539989    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx" pod "38d3bbfc-f466-49d3-b876-910aa65efd6a" (UID: "38d3bbfc-f466-49d3-b876-910aa65efd6a"). InnerVolumeSpecName "kube-api-access-ddddx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:06.627559 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627527    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddddx\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-kube-api-access-ddddx\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627559 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627559    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627570    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627579    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627587    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627596    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627603    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627611    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d3bbfc-f466-49d3-b876-910aa65efd6a-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627621    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ck6\" (UniqueName: \"kubernetes.io/projected/6e61eebf-fb19-4f5f-80ae-9bd904347052-kube-api-access-c4ck6\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627629    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e61eebf-fb19-4f5f-80ae-9bd904347052-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:06.627772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:06.627638    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d3bbfc-f466-49d3-b876-910aa65efd6a-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.389264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.389230    6167 generic.go:356] "Generic (PLEG): container finished" podID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerID="a21346ba92ab4667edbe008dacbf8934aaaa5fe23de8565c14b2ff12ebd9e03f" exitCode=0
Nov 03 18:16:07.389748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.389393    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" event={"ID":"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a","Type":"ContainerDied","Data":"a21346ba92ab4667edbe008dacbf8934aaaa5fe23de8565c14b2ff12ebd9e03f"}
Nov 03 18:16:07.391299 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.391264    6167 generic.go:356] "Generic (PLEG): container finished" podID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerID="17a0efcbfb837a885d8c001ad03634508a3d74b7202852a87f9d01ebf505f215" exitCode=0
Nov 03 18:16:07.391433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.391336    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" event={"ID":"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a","Type":"ContainerDied","Data":"17a0efcbfb837a885d8c001ad03634508a3d74b7202852a87f9d01ebf505f215"}
Nov 03 18:16:07.393736 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.392864    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt" event={"ID":"6e61eebf-fb19-4f5f-80ae-9bd904347052","Type":"ContainerDied","Data":"ab7e6306e1a8397a3b2b1c42235b22d7fcbcc8d3c98c8ad589148750b331ffcb"}
Nov 03 18:16:07.393736 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.392913    6167 scope.go:122] "RemoveContainer" containerID="33947b4d0cf5fe421be106e8693eb4385c6675b24aa4fa839ce6697d60af9e20"
Nov 03 18:16:07.393736 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.393034    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"
Nov 03 18:16:07.396244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.395585    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42" event={"ID":"38d3bbfc-f466-49d3-b876-910aa65efd6a","Type":"ContainerDied","Data":"4c61e77a3caf7546d285aa5b582f13bc4a18f123eb544740d589d1ede1d41df0"}
Nov 03 18:16:07.396244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.395673    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"
Nov 03 18:16:07.406913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.406824    6167 scope.go:122] "RemoveContainer" containerID="4a67a8e20f39b7f93dd251b412ebcd1f6ca98c21ec69f23aae06d091754c1b1b"
Nov 03 18:16:07.485936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.485905    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:16:07.538414 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538382    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538553 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538504    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538631 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538602    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538664    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538710    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538777    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538817    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538953 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538871    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.538953 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.538944    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy\") pod \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\" (UID: \"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a\") "
Nov 03 18:16:07.539056 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539029    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.539258 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539239    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.539524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539500    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.539592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539522    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.539592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539541    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.539803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539779    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:07.539923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.539834    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.541047 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.541012    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "kube-api-access-k7xf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:07.541137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.541099    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:07.541482 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.541457    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.541829 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.541809    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token" pod "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" (UID: "b89fdd5f-25ed-4b9f-9a77-56c170f9c17a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:07.544891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.544873    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:16:07.613511 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.613460    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"]
Nov 03 18:16:07.617942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.617910    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-6db42"]
Nov 03 18:16:07.640585 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640552    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.640725 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640617    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.640725 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640668    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.640725 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640701    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.640907 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640833    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.640907 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640872    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.641018 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640907    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.641018 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.640987    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.641134 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641028    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.641134 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641056    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket\") pod \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\" (UID: \"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a\") "
Nov 03 18:16:07.641276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641249    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.641353 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641279    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:07.641412 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641386    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.641514 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641487    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641712    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7xf2\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-kube-api-access-k7xf2\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641763    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641774    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641785    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641793    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641802    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641811    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641820    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641839    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641852    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641861    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.641898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.641875    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.643447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.643427    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:16:07.643587 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.643568    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:07.643679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.643655    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:07.643828 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.643807    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg" pod "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" (UID: "a424743b-e4c8-4e6e-ad03-5edf1d6fe60a"). InnerVolumeSpecName "kube-api-access-2btsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:07.742831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.742746    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btsg\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-kube-api-access-2btsg\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.742831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.742786    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.742831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.742801    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.742831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.742815    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:07.850632 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.850596    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"]
Nov 03 18:16:07.854799 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:07.854768    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-mfdrt"]
Nov 03 18:16:08.101821 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.101787    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"]
Nov 03 18:16:08.203350 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.203321    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"]
Nov 03 18:16:08.408273 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.408192    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"
Nov 03 18:16:08.408273 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.408232    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj" event={"ID":"b89fdd5f-25ed-4b9f-9a77-56c170f9c17a","Type":"ContainerDied","Data":"2efbc76655307b74930b7850c7e3f798e54231fd0145b03a964be43074c6760d"}
Nov 03 18:16:08.408730 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.408302    6167 scope.go:122] "RemoveContainer" containerID="a21346ba92ab4667edbe008dacbf8934aaaa5fe23de8565c14b2ff12ebd9e03f"
Nov 03 18:16:08.416748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.413586    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr" event={"ID":"a424743b-e4c8-4e6e-ad03-5edf1d6fe60a","Type":"ContainerDied","Data":"efb00a55382529a0fc7b3ca2b541fb74b4cfaa1e1c935975853ca1fe4c2ca177"}
Nov 03 18:16:08.416748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.413679    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"
Nov 03 18:16:08.430651 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.430632    6167 scope.go:122] "RemoveContainer" containerID="17a0efcbfb837a885d8c001ad03634508a3d74b7202852a87f9d01ebf505f215"
Nov 03 18:16:08.577184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.577130    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"]
Nov 03 18:16:08.580489 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.580463    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-fqqcj"]
Nov 03 18:16:08.691064 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.690987    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"]
Nov 03 18:16:08.694969 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.694940    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-nl7pr"]
Nov 03 18:16:08.929055 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.929017    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" path="/var/lib/kubelet/pods/38d3bbfc-f466-49d3-b876-910aa65efd6a/volumes"
Nov 03 18:16:08.929824 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.929795    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" path="/var/lib/kubelet/pods/6e61eebf-fb19-4f5f-80ae-9bd904347052/volumes"
Nov 03 18:16:08.930458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.930440    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" path="/var/lib/kubelet/pods/a424743b-e4c8-4e6e-ad03-5edf1d6fe60a/volumes"
Nov 03 18:16:08.931528 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:08.931508    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" path="/var/lib/kubelet/pods/b89fdd5f-25ed-4b9f-9a77-56c170f9c17a/volumes"
Nov 03 18:16:09.291752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.291634    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"]
Nov 03 18:16:09.292741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.292648    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w" podUID="8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" containerName="maas-api" containerID="cri-o://d079b8453a3d3a493e1ac0931845a227e5c63d1a751a0dd766a3ad5e912cdffe" gracePeriod=30
Nov 03 18:16:09.440115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.440069    6167 generic.go:356] "Generic (PLEG): container finished" podID="8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" containerID="d079b8453a3d3a493e1ac0931845a227e5c63d1a751a0dd766a3ad5e912cdffe" exitCode=0
Nov 03 18:16:09.440549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.440142    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w" event={"ID":"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d","Type":"ContainerDied","Data":"d079b8453a3d3a493e1ac0931845a227e5c63d1a751a0dd766a3ad5e912cdffe"}
Nov 03 18:16:09.474286 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.474255    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:16:09.608281 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.608249    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:16:09.697542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.697506    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs\") pod \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") "
Nov 03 18:16:09.697730 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.697552    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca\") pod \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") "
Nov 03 18:16:09.697730 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.697706    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7\" (UniqueName: \"kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7\") pod \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") "
Nov 03 18:16:09.697867 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.697771    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\") pod \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\" (UID: \"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d\") "
Nov 03 18:16:09.697968 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.697939    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca" pod "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" (UID: "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:09.698408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.698383    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:09.700483 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.700451    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7" pod "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" (UID: "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d"). InnerVolumeSpecName "kube-api-access-fsbh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:09.700483 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.700475    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls" pod "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" (UID: "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:09.700965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.700943    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs" pod "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" (UID: "8a4a62d6-14de-4a05-a56d-ca5cdb832e8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:09.801581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.801501    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbh7\" (UniqueName: \"kubernetes.io/projected/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-kube-api-access-fsbh7\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:09.801581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.801545    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:09.801581 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:09.801579    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.349567 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.349530    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"]
Nov 03 18:16:10.349823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.349766    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" podUID="596176e2-4358-4834-864e-d12afa8f0282" containerName="maas-api" containerID="cri-o://54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf" gracePeriod=30
Nov 03 18:16:10.452900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.452870    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"
Nov 03 18:16:10.453370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.452909    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w" event={"ID":"8a4a62d6-14de-4a05-a56d-ca5cdb832e8d","Type":"ContainerDied","Data":"5619ea6d5eac622d87b1f3da9aac71a7835237996b0412e1afbdf8d14769f902"}
Nov 03 18:16:10.453370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.452975    6167 scope.go:122] "RemoveContainer" containerID="d079b8453a3d3a493e1ac0931845a227e5c63d1a751a0dd766a3ad5e912cdffe"
Nov 03 18:16:10.589019 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.588976    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"]
Nov 03 18:16:10.599524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.599491    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-trs9w"]
Nov 03 18:16:10.642340 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.642314    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:16:10.712794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.712762    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n\" (UniqueName: \"kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n\") pod \"596176e2-4358-4834-864e-d12afa8f0282\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") "
Nov 03 18:16:10.712991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.712809    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\") pod \"596176e2-4358-4834-864e-d12afa8f0282\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") "
Nov 03 18:16:10.712991 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.712958    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca\") pod \"596176e2-4358-4834-864e-d12afa8f0282\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") "
Nov 03 18:16:10.713123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.713032    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs\") pod \"596176e2-4358-4834-864e-d12afa8f0282\" (UID: \"596176e2-4358-4834-864e-d12afa8f0282\") "
Nov 03 18:16:10.716102 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.716072    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls" pod "596176e2-4358-4834-864e-d12afa8f0282" (UID: "596176e2-4358-4834-864e-d12afa8f0282"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:10.727122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.727063    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca" pod "596176e2-4358-4834-864e-d12afa8f0282" (UID: "596176e2-4358-4834-864e-d12afa8f0282"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:10.740777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.736167    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n" pod "596176e2-4358-4834-864e-d12afa8f0282" (UID: "596176e2-4358-4834-864e-d12afa8f0282"). InnerVolumeSpecName "kube-api-access-mlx2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:10.771945 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.769257    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs" pod "596176e2-4358-4834-864e-d12afa8f0282" (UID: "596176e2-4358-4834-864e-d12afa8f0282"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:10.814368 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.814328    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlx2n\" (UniqueName: \"kubernetes.io/projected/596176e2-4358-4834-864e-d12afa8f0282-kube-api-access-mlx2n\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.814368 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.814365    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.814368 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.814375    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/596176e2-4358-4834-864e-d12afa8f0282-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.814607 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.814384    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596176e2-4358-4834-864e-d12afa8f0282-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:10.929748 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:10.929657    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" path="/var/lib/kubelet/pods/8a4a62d6-14de-4a05-a56d-ca5cdb832e8d/volumes"
Nov 03 18:16:11.462469 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.462437    6167 generic.go:356] "Generic (PLEG): container finished" podID="596176e2-4358-4834-864e-d12afa8f0282" containerID="54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf" exitCode=0
Nov 03 18:16:11.462917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.462497    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"
Nov 03 18:16:11.462917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.462503    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" event={"ID":"596176e2-4358-4834-864e-d12afa8f0282","Type":"ContainerDied","Data":"54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf"}
Nov 03 18:16:11.462917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.462615    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6" event={"ID":"596176e2-4358-4834-864e-d12afa8f0282","Type":"ContainerDied","Data":"50de294fad6328c2d408723df7826c5e728a219a70331de34d52e967bbdb0c10"}
Nov 03 18:16:11.462917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.462636    6167 scope.go:122] "RemoveContainer" containerID="54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf"
Nov 03 18:16:11.472618 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.472586    6167 scope.go:122] "RemoveContainer" containerID="54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf"
Nov 03 18:16:11.472878 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:16:11.472860    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf\": container with ID starting with 54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf not found: ID does not exist" containerID="54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf"
Nov 03 18:16:11.472934 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.472885    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf"} err="failed to get container status \"54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf\": rpc error: code = NotFound desc = could not find container \"54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf\": container with ID starting with 54c685fbd9c72c04552d8d8e5475af72122415741b773b648486adee551a6faf not found: ID does not exist"
Nov 03 18:16:11.630248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.630213    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"]
Nov 03 18:16:11.680535 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:11.680507    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-80aca1-76577b575-xrsn6"]
Nov 03 18:16:12.928794 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:12.928763    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596176e2-4358-4834-864e-d12afa8f0282" path="/var/lib/kubelet/pods/596176e2-4358-4834-864e-d12afa8f0282/volumes"
Nov 03 18:16:13.102360 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.102319    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm" podUID="4de76cfd-889f-4565-9f60-3434798f70d7" containerName="payload-processing" containerID="cri-o://85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270" gracePeriod=25
Nov 03 18:16:13.204532 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.204427    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw" podUID="5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" containerName="payload-pre-processing" containerID="cri-o://6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf" gracePeriod=25
Nov 03 18:16:13.381845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.381811    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:16:13.447024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.446987    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume\") pod \"4de76cfd-889f-4565-9f60-3434798f70d7\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") "
Nov 03 18:16:13.447189 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.447156    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq\" (UniqueName: \"kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq\") pod \"4de76cfd-889f-4565-9f60-3434798f70d7\" (UID: \"4de76cfd-889f-4565-9f60-3434798f70d7\") "
Nov 03 18:16:13.447421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.447394    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume" pod "4de76cfd-889f-4565-9f60-3434798f70d7" (UID: "4de76cfd-889f-4565-9f60-3434798f70d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:13.447636 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.447618    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de76cfd-889f-4565-9f60-3434798f70d7-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:13.449251 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.449222    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq" pod "4de76cfd-889f-4565-9f60-3434798f70d7" (UID: "4de76cfd-889f-4565-9f60-3434798f70d7"). InnerVolumeSpecName "kube-api-access-b7zcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:13.474845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.474821    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:16:13.478230 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.478206    6167 generic.go:356] "Generic (PLEG): container finished" podID="5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" containerID="6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf" exitCode=0
Nov 03 18:16:13.478332 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.478234    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"
Nov 03 18:16:13.478332 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.478288    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw" event={"ID":"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7","Type":"ContainerDied","Data":"6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf"}
Nov 03 18:16:13.478407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.478340    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw" event={"ID":"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7","Type":"ContainerDied","Data":"37753eda0452ca6f1dbe7add225cbdae0c62315c5ce44e2926eb5fb0a1d20832"}
Nov 03 18:16:13.478407 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.478364    6167 scope.go:122] "RemoveContainer" containerID="6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf"
Nov 03 18:16:13.479790 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.479751    6167 generic.go:356] "Generic (PLEG): container finished" podID="4de76cfd-889f-4565-9f60-3434798f70d7" containerID="85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270" exitCode=0
Nov 03 18:16:13.479790 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.479779    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm" event={"ID":"4de76cfd-889f-4565-9f60-3434798f70d7","Type":"ContainerDied","Data":"85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270"}
Nov 03 18:16:13.479941 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.479807    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"
Nov 03 18:16:13.479941 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.479837    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm" event={"ID":"4de76cfd-889f-4565-9f60-3434798f70d7","Type":"ContainerDied","Data":"d6496d2e6b0b3a28dc82b9aa83dfa9c5ab165c7c1fcf78a396b4a543a8d887ff"}
Nov 03 18:16:13.488539 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.488515    6167 scope.go:122] "RemoveContainer" containerID="6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf"
Nov 03 18:16:13.488824 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:16:13.488801    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf\": container with ID starting with 6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf not found: ID does not exist" containerID="6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf"
Nov 03 18:16:13.488891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.488832    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf"} err="failed to get container status \"6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf\": rpc error: code = NotFound desc = could not find container \"6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf\": container with ID starting with 6a7b826bb98148e71be4ddd33de9814c421d544ca11a3dcf3d753f1177e49adf not found: ID does not exist"
Nov 03 18:16:13.488891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.488848    6167 scope.go:122] "RemoveContainer" containerID="85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270"
Nov 03 18:16:13.498194 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.498180    6167 scope.go:122] "RemoveContainer" containerID="85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270"
Nov 03 18:16:13.498490 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:16:13.498470    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270\": container with ID starting with 85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270 not found: ID does not exist" containerID="85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270"
Nov 03 18:16:13.498583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.498495    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270"} err="failed to get container status \"85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270\": rpc error: code = NotFound desc = could not find container \"85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270\": container with ID starting with 85467664b9b27fa2477aa96b67a43d6b07b8bca78d7dba9b2d04205dcc52b270 not found: ID does not exist"
Nov 03 18:16:13.548824 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.548790    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume\") pod \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") "
Nov 03 18:16:13.548990 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.548879    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs\" (UniqueName: \"kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs\") pod \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\" (UID: \"5b856b0a-29bb-4395-8c86-ae5a0e50eaf7\") "
Nov 03 18:16:13.549254 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.549230    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume" pod "5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" (UID: "5b856b0a-29bb-4395-8c86-ae5a0e50eaf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:13.549312 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.549299    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zcq\" (UniqueName: \"kubernetes.io/projected/4de76cfd-889f-4565-9f60-3434798f70d7-kube-api-access-b7zcq\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:13.550948 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.550926    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs" pod "5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" (UID: "5b856b0a-29bb-4395-8c86-ae5a0e50eaf7"). InnerVolumeSpecName "kube-api-access-gx9zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:13.651014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.650982    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9zs\" (UniqueName: \"kubernetes.io/projected/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-kube-api-access-gx9zs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:13.651014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.651010    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:13.732159 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.732062    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"]
Nov 03 18:16:13.737432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.737399    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-80aca1-58bb6b8bff-w4rdm"]
Nov 03 18:16:13.902557 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.902518    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"]
Nov 03 18:16:13.907520 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:13.907495    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-80aca1-b7f99bbc-rkdkw"]
Nov 03 18:16:14.928780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:14.928748    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de76cfd-889f-4565-9f60-3434798f70d7" path="/var/lib/kubelet/pods/4de76cfd-889f-4565-9f60-3434798f70d7/volumes"
Nov 03 18:16:14.929182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:14.929168    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" path="/var/lib/kubelet/pods/5b856b0a-29bb-4395-8c86-ae5a0e50eaf7/volumes"
Nov 03 18:16:24.522279 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:16:24.522238    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:16:34.840402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:34.840366    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"]
Nov 03 18:16:34.841064 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:34.840552    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" podUID="d1a2a820-3ce2-4afb-808e-f8378d7540ed" containerName="maas-api" containerID="cri-o://2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6" gracePeriod=30
Nov 03 18:16:35.132067 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.132037    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:16:35.218602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.218570    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs\") pod \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") "
Nov 03 18:16:35.218780 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.218670    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4\" (UniqueName: \"kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4\") pod \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") "
Nov 03 18:16:35.218894 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.218877    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\") pod \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") "
Nov 03 18:16:35.218953 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.218937    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca\") pod \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\" (UID: \"d1a2a820-3ce2-4afb-808e-f8378d7540ed\") "
Nov 03 18:16:35.219408 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.219362    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca" pod "d1a2a820-3ce2-4afb-808e-f8378d7540ed" (UID: "d1a2a820-3ce2-4afb-808e-f8378d7540ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:35.220854 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.220827    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs" pod "d1a2a820-3ce2-4afb-808e-f8378d7540ed" (UID: "d1a2a820-3ce2-4afb-808e-f8378d7540ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:35.220971 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.220901    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls" pod "d1a2a820-3ce2-4afb-808e-f8378d7540ed" (UID: "d1a2a820-3ce2-4afb-808e-f8378d7540ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:35.220971 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.220909    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4" pod "d1a2a820-3ce2-4afb-808e-f8378d7540ed" (UID: "d1a2a820-3ce2-4afb-808e-f8378d7540ed"). InnerVolumeSpecName "kube-api-access-ccpv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:35.321026 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.320989    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:35.321026 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.321020    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a2a820-3ce2-4afb-808e-f8378d7540ed-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:35.321026 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.321032    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a2a820-3ce2-4afb-808e-f8378d7540ed-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:35.321274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.321042    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpv4\" (UniqueName: \"kubernetes.io/projected/d1a2a820-3ce2-4afb-808e-f8378d7540ed-kube-api-access-ccpv4\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:16:35.646359 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.646319    6167 generic.go:356] "Generic (PLEG): container finished" podID="d1a2a820-3ce2-4afb-808e-f8378d7540ed" containerID="2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6" exitCode=0
Nov 03 18:16:35.646556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.646383    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"
Nov 03 18:16:35.646556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.646414    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" event={"ID":"d1a2a820-3ce2-4afb-808e-f8378d7540ed","Type":"ContainerDied","Data":"2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6"}
Nov 03 18:16:35.646556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.646493    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp" event={"ID":"d1a2a820-3ce2-4afb-808e-f8378d7540ed","Type":"ContainerDied","Data":"360ed44c4d06f00a253176b973bef14156e28a2b378123589560e7ffff91c0e4"}
Nov 03 18:16:35.646556 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.646515    6167 scope.go:122] "RemoveContainer" containerID="2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6"
Nov 03 18:16:35.656839 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.656824    6167 scope.go:122] "RemoveContainer" containerID="2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6"
Nov 03 18:16:35.657111 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:16:35.657091    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6\": container with ID starting with 2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6 not found: ID does not exist" containerID="2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6"
Nov 03 18:16:35.657173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.657119    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6"} err="failed to get container status \"2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6\": rpc error: code = NotFound desc = could not find container \"2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6\": container with ID starting with 2731883ede0085632ad917ddcbbd331d78941c74668d48d373a20ff6243d8fc6 not found: ID does not exist"
Nov 03 18:16:35.813733 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.813692    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"]
Nov 03 18:16:35.817327 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:35.817302    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c1b2d2-684b8878bf-pd8fp"]
Nov 03 18:16:36.360767 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.360728    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"]
Nov 03 18:16:36.361443 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361420    6167 cpu_manager.go:414] "containerMap: removing container" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361444    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21faf221-c045-4d19-88ca-6a2fd72667be" containerName="istio-proxy"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361469    6167 cpu_manager.go:414] "containerMap: removing container" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerName="payload-pre-processing"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361477    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerName="payload-pre-processing"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361491    6167 cpu_manager.go:414] "containerMap: removing container" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361499    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361512    6167 cpu_manager.go:414] "containerMap: removing container" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361519    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361530    6167 cpu_manager.go:414] "containerMap: removing container" podUID="a912bc62-c39d-4a84-8092-055d55a2f59f" containerName="payload-processing"
Nov 03 18:16:36.361543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361537    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a912bc62-c39d-4a84-8092-055d55a2f59f" containerName="payload-processing"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361551    6167 cpu_manager.go:414] "containerMap: removing container" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361558    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361573    6167 cpu_manager.go:414] "containerMap: removing container" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361580    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361592    6167 cpu_manager.go:414] "containerMap: removing container" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361599    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361611    6167 cpu_manager.go:414] "containerMap: removing container" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361618    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361634    6167 cpu_manager.go:414] "containerMap: removing container" podUID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361641    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361650    6167 cpu_manager.go:414] "containerMap: removing container" podUID="4de76cfd-889f-4565-9f60-3434798f70d7" containerName="payload-processing"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361657    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de76cfd-889f-4565-9f60-3434798f70d7" containerName="payload-processing"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361671    6167 cpu_manager.go:414] "containerMap: removing container" podUID="b2db6fdf-7669-4ed8-9299-4100c1850311" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361679    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2db6fdf-7669-4ed8-9299-4100c1850311" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361691    6167 cpu_manager.go:414] "containerMap: removing container" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361698    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361729    6167 cpu_manager.go:414] "containerMap: removing container" podUID="d1a2a820-3ce2-4afb-808e-f8378d7540ed" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361737    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a2a820-3ce2-4afb-808e-f8378d7540ed" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361752    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361759    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361773    6167 cpu_manager.go:414] "containerMap: removing container" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361780    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361793    6167 cpu_manager.go:414] "containerMap: removing container" podUID="89064973-1a13-4dca-80b5-6e9ee9c1d9fa" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361801    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89064973-1a13-4dca-80b5-6e9ee9c1d9fa" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361821    6167 cpu_manager.go:414] "containerMap: removing container" podUID="596176e2-4358-4834-864e-d12afa8f0282" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361828    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596176e2-4358-4834-864e-d12afa8f0282" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361837    6167 cpu_manager.go:414] "containerMap: removing container" podUID="5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" containerName="payload-pre-processing"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361852    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" containerName="payload-pre-processing"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361865    6167 cpu_manager.go:414] "containerMap: removing container" podUID="8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361872    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361888    6167 cpu_manager.go:414] "containerMap: removing container" podUID="a1a6f824-e9f5-4f47-a8f2-74b4e266c174" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.361895    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a6f824-e9f5-4f47-a8f2-74b4e266c174" containerName="maas-api"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362034    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6df85b-f8b8-4f2f-927c-e0f4e5e99dc1" containerName="istio-proxy"
Nov 03 18:16:36.362028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362047    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362058    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="89064973-1a13-4dca-80b5-6e9ee9c1d9fa" containerName="maas-api"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362070    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="596176e2-4358-4834-864e-d12afa8f0282" containerName="maas-api"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362080    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b856b0a-29bb-4395-8c86-ae5a0e50eaf7" containerName="payload-pre-processing"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362088    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4a62d6-14de-4a05-a56d-ca5cdb832e8d" containerName="maas-api"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362099    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a6f824-e9f5-4f47-a8f2-74b4e266c174" containerName="maas-api"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362110    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad2dad4-78e8-4a6f-81f9-1c004379ee3a" containerName="payload-pre-processing"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362119    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5272092-0dea-4ffd-93e1-e64256cffbd5" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362128    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="a424743b-e4c8-4e6e-ad03-5edf1d6fe60a" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362136    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="a912bc62-c39d-4a84-8092-055d55a2f59f" containerName="payload-processing"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362144    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89fdd5f-25ed-4b9f-9a77-56c170f9c17a" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362153    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e61eebf-fb19-4f5f-80ae-9bd904347052" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362163    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="5501f821-1679-48b9-9bde-4dbaaf0450fb" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362173    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="207b02f4-b43d-4c8a-9dd8-fd77189406dc" containerName="maas-api"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362183    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de76cfd-889f-4565-9f60-3434798f70d7" containerName="payload-processing"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362191    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2db6fdf-7669-4ed8-9299-4100c1850311" containerName="maas-api"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362201    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40f64c2-53b5-4d1d-9f9d-b9207429f502" containerName="istio-proxy"
Nov 03 18:16:36.363678 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.362211    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a2a820-3ce2-4afb-808e-f8378d7540ed" containerName="maas-api"
Nov 03 18:16:36.367049 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.367030    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.425645 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.425608    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"]
Nov 03 18:16:36.434645 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.434612    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmjk\" (UniqueName: \"kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.434835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.434665    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.434835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.434803    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.434963 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.434866    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.449417 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.449389    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1720c00c-serving-cert\""
Nov 03 18:16:36.535891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.535855    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmjk\" (UniqueName: \"kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.536081 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.535897    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.536081 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.535929    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.536081 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.535969    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.536643 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.536619    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.538863 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.538839    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.538863 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.538857    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.598076 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.598040    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmjk\" (UniqueName: \"kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk\") pod \"maas-api-e2e-mt-1720c00c-6977b66858-gdw82\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.680862 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.680758    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:36.928679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:36.928648    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a2a820-3ce2-4afb-808e-f8378d7540ed" path="/var/lib/kubelet/pods/d1a2a820-3ce2-4afb-808e-f8378d7540ed/volumes"
Nov 03 18:16:37.058779 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:16:37.058746    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f11d44_9763_4901_af47_ea7a5a7d0697.slice/crio-0fe9a688641b6b8bb467770ff63d40252f217cf95a0a1acb9282fe25549e189a WatchSource:0}: Error finding container 0fe9a688641b6b8bb467770ff63d40252f217cf95a0a1acb9282fe25549e189a: Status 404 returned error can't find the container with id 0fe9a688641b6b8bb467770ff63d40252f217cf95a0a1acb9282fe25549e189a
Nov 03 18:16:37.061531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:37.061508    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"]
Nov 03 18:16:37.665815 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:37.665770    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82" event={"ID":"d3f11d44-9763-4901-af47-ea7a5a7d0697","Type":"ContainerStarted","Data":"0fe9a688641b6b8bb467770ff63d40252f217cf95a0a1acb9282fe25549e189a"}
Nov 03 18:16:38.676264 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:38.676225    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82" event={"ID":"d3f11d44-9763-4901-af47-ea7a5a7d0697","Type":"ContainerStarted","Data":"e9c82b23035e27e9854103f8e7fa4d2c8104fac00585ed52e180aba0e0cacf72"}
Nov 03 18:16:38.676764 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:38.676300    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:38.799339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:38.799277    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82" podStartSLOduration=2.002995649 podStartE2EDuration="2.799262627s" totalImagesPullingTime="796.266978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:38.798888915 +0000 UTC m=+1854.029213987" watchObservedRunningTime="2026-11-03 18:16:38.799262627 +0000 UTC m=+1854.029587693"
Nov 03 18:16:44.688026 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:44.687996    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:16:48.438201 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:48.438164    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:16:49.316575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:16:49.316538    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"]
Nov 03 18:16:54.520419 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:16:54.520383    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:17:04.339410 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.339337    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main" containerID="cri-o://04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca" gracePeriod=45
Nov 03 18:17:04.619255 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.619228    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:17:04.676300 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676261    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir\") pod \"0214d7cf-6f58-4889-99d8-f98e8553a400\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") "
Nov 03 18:17:04.676448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676378    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm\") pod \"0214d7cf-6f58-4889-99d8-f98e8553a400\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") "
Nov 03 18:17:04.676448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676426    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb\" (UniqueName: \"kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb\") pod \"0214d7cf-6f58-4889-99d8-f98e8553a400\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") "
Nov 03 18:17:04.676566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676458    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs\" (UniqueName: \"kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs\") pod \"0214d7cf-6f58-4889-99d8-f98e8553a400\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") "
Nov 03 18:17:04.676566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676524    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home\") pod \"0214d7cf-6f58-4889-99d8-f98e8553a400\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") "
Nov 03 18:17:04.676712 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676581    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir" pod "0214d7cf-6f58-4889-99d8-f98e8553a400" (UID: "0214d7cf-6f58-4889-99d8-f98e8553a400"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.676712 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676608    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache\") pod \"0214d7cf-6f58-4889-99d8-f98e8553a400\" (UID: \"0214d7cf-6f58-4889-99d8-f98e8553a400\") "
Nov 03 18:17:04.676848 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.676815    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home" pod "0214d7cf-6f58-4889-99d8-f98e8553a400" (UID: "0214d7cf-6f58-4889-99d8-f98e8553a400"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.677053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.677022    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache" pod "0214d7cf-6f58-4889-99d8-f98e8553a400" (UID: "0214d7cf-6f58-4889-99d8-f98e8553a400"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.677229 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.677210    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.677305 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.677235    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.677305 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.677249    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.678843 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.678795    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs" pod "0214d7cf-6f58-4889-99d8-f98e8553a400" (UID: "0214d7cf-6f58-4889-99d8-f98e8553a400"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:04.678843 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.678800    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb" pod "0214d7cf-6f58-4889-99d8-f98e8553a400" (UID: "0214d7cf-6f58-4889-99d8-f98e8553a400"). InnerVolumeSpecName "kube-api-access-f5wcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:04.678972 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.678895    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm" pod "0214d7cf-6f58-4889-99d8-f98e8553a400" (UID: "0214d7cf-6f58-4889-99d8-f98e8553a400"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:04.778220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.778186    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0214d7cf-6f58-4889-99d8-f98e8553a400-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.778220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.778217    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5wcb\" (UniqueName: \"kubernetes.io/projected/0214d7cf-6f58-4889-99d8-f98e8553a400-kube-api-access-f5wcb\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.778220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.778227    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0214d7cf-6f58-4889-99d8-f98e8553a400-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:04.889072 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.888986    6167 generic.go:356] "Generic (PLEG): container finished" podID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerID="04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca" exitCode=0
Nov 03 18:17:04.889247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.889097    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"
Nov 03 18:17:04.889247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.889126    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" event={"ID":"0214d7cf-6f58-4889-99d8-f98e8553a400","Type":"ContainerDied","Data":"04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca"}
Nov 03 18:17:04.889247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.889192    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh" event={"ID":"0214d7cf-6f58-4889-99d8-f98e8553a400","Type":"ContainerDied","Data":"6b0a341f79021742d0dbe23ec7cb0e7d1fa081d4c6adf92d2bd1fbe8f76c114b"}
Nov 03 18:17:04.889247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.889220    6167 scope.go:122] "RemoveContainer" containerID="04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca"
Nov 03 18:17:04.901198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.901172    6167 scope.go:122] "RemoveContainer" containerID="04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca"
Nov 03 18:17:04.901549 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:17:04.901526    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca\": container with ID starting with 04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca not found: ID does not exist" containerID="04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca"
Nov 03 18:17:04.901657 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:04.901555    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca"} err="failed to get container status \"04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca\": rpc error: code = NotFound desc = could not find container \"04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca\": container with ID starting with 04431ae4fa0db321e13c1d7058530ed15aff2822cc94e26a18830b300166f6ca not found: ID does not exist"
Nov 03 18:17:05.059349 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:05.059308    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"]
Nov 03 18:17:05.065771 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:05.065741    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/distinct-2-w2-f79c02-kserve-7c67d6bd6f-69mvh"]
Nov 03 18:17:06.927562 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:06.927482    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" path="/var/lib/kubelet/pods/0214d7cf-6f58-4889-99d8-f98e8553a400/volumes"
Nov 03 18:17:14.751941 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:14.751906    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"]
Nov 03 18:17:14.752356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:14.752219    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" podUID="11f7da85-fbe3-47ec-8bb5-4e138ac94e86" containerName="maas-api" containerID="cri-o://5cb0ad5b465af6258e2a5b4ef397d2574c0bca0caf34173a7f251acfd9ef83c3" gracePeriod=30
Nov 03 18:17:14.959028 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:14.958989    6167 generic.go:356] "Generic (PLEG): container finished" podID="11f7da85-fbe3-47ec-8bb5-4e138ac94e86" containerID="5cb0ad5b465af6258e2a5b4ef397d2574c0bca0caf34173a7f251acfd9ef83c3" exitCode=0
Nov 03 18:17:14.959221 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:14.959061    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" event={"ID":"11f7da85-fbe3-47ec-8bb5-4e138ac94e86","Type":"ContainerDied","Data":"5cb0ad5b465af6258e2a5b4ef397d2574c0bca0caf34173a7f251acfd9ef83c3"}
Nov 03 18:17:15.056566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.056537    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:17:15.210801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.210768    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca\") pod \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") "
Nov 03 18:17:15.210986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.210855    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs\") pod \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") "
Nov 03 18:17:15.210986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.210909    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\") pod \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") "
Nov 03 18:17:15.210986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.210957    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5\" (UniqueName: \"kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5\") pod \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\" (UID: \"11f7da85-fbe3-47ec-8bb5-4e138ac94e86\") "
Nov 03 18:17:15.211207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.211172    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca" pod "11f7da85-fbe3-47ec-8bb5-4e138ac94e86" (UID: "11f7da85-fbe3-47ec-8bb5-4e138ac94e86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:15.211532 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.211508    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:15.213213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.213183    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5" pod "11f7da85-fbe3-47ec-8bb5-4e138ac94e86" (UID: "11f7da85-fbe3-47ec-8bb5-4e138ac94e86"). InnerVolumeSpecName "kube-api-access-78gb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:15.213558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.213535    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls" pod "11f7da85-fbe3-47ec-8bb5-4e138ac94e86" (UID: "11f7da85-fbe3-47ec-8bb5-4e138ac94e86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:15.213658 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.213519    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs" pod "11f7da85-fbe3-47ec-8bb5-4e138ac94e86" (UID: "11f7da85-fbe3-47ec-8bb5-4e138ac94e86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:15.313333 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.313230    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:15.313333 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.313273    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:15.313333 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.313288    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78gb5\" (UniqueName: \"kubernetes.io/projected/11f7da85-fbe3-47ec-8bb5-4e138ac94e86-kube-api-access-78gb5\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:15.985742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.969077    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs" event={"ID":"11f7da85-fbe3-47ec-8bb5-4e138ac94e86","Type":"ContainerDied","Data":"bbb4e51d98ef8cb06223e1ee6d84b8abbe815ff2b974624c216dce0de4440b8b"}
Nov 03 18:17:15.985742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.969156    6167 scope.go:122] "RemoveContainer" containerID="5cb0ad5b465af6258e2a5b4ef397d2574c0bca0caf34173a7f251acfd9ef83c3"
Nov 03 18:17:15.985742 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:15.969327    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"
Nov 03 18:17:16.138964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:16.138927    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"]
Nov 03 18:17:16.189555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:16.189520    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-232f9e-767bb959c7-fcrzs"]
Nov 03 18:17:16.927840 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:16.927807    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f7da85-fbe3-47ec-8bb5-4e138ac94e86" path="/var/lib/kubelet/pods/11f7da85-fbe3-47ec-8bb5-4e138ac94e86/volumes"
Nov 03 18:17:24.520693 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:17:24.520613    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:17:29.983413 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:29.983376    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"]
Nov 03 18:17:30.261775 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.261688    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"]
Nov 03 18:17:30.262009 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.261985    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" podUID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" containerName="maas-api" containerID="cri-o://0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c" gracePeriod=30
Nov 03 18:17:30.560499 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.560471    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 18:17:30.595681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.595653    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca\") pod \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") "
Nov 03 18:17:30.595868 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.595858    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t\" (UniqueName: \"kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t\") pod \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") "
Nov 03 18:17:30.595913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.595883    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs\") pod \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") "
Nov 03 18:17:30.595987 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.595972    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls\") pod \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\" (UID: \"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4\") "
Nov 03 18:17:30.596023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.595998    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca" pod "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" (UID: "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:30.596439 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.596413    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:30.598056 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.598028    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs" pod "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" (UID: "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:30.598174 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.598093    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t" pod "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" (UID: "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4"). InnerVolumeSpecName "kube-api-access-w249t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:30.598174 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.598159    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls" pod "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" (UID: "8ab055d1-5a28-41cf-aea7-e1fb050e8ee4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:30.697384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.697346    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w249t\" (UniqueName: \"kubernetes.io/projected/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-kube-api-access-w249t\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:30.697634 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.697605    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:30.697634 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:30.697635    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:31.073179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.073146    6167 generic.go:356] "Generic (PLEG): container finished" podID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" containerID="0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c" exitCode=0
Nov 03 18:17:31.073647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.073203    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"
Nov 03 18:17:31.073647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.073223    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" event={"ID":"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4","Type":"ContainerDied","Data":"0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c"}
Nov 03 18:17:31.073647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.073291    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" event={"ID":"8ab055d1-5a28-41cf-aea7-e1fb050e8ee4","Type":"ContainerDied","Data":"6cbc098206d067fadbeae85ccfaeb46e92691db282a06b3091f58289139ec4b7"}
Nov 03 18:17:31.073647 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.073326    6167 scope.go:122] "RemoveContainer" containerID="0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c"
Nov 03 18:17:31.084690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.084671    6167 scope.go:122] "RemoveContainer" containerID="0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c"
Nov 03 18:17:31.084979 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:17:31.084960    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c\": container with ID starting with 0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c not found: ID does not exist" containerID="0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c"
Nov 03 18:17:31.085030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.084988    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c"} err="failed to get container status \"0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c\": rpc error: code = NotFound desc = could not find container \"0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c\": container with ID starting with 0bb5afe5f196f624550367ddc89b74eb9bdf7360723bd49dc983f796bb493b9c not found: ID does not exist"
Nov 03 18:17:31.240292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.240207    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"]
Nov 03 18:17:31.290354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:31.290324    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm"]
Nov 03 18:17:32.280507 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:32.280471    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:17:32.929461 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:32.929430    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" path="/var/lib/kubelet/pods/8ab055d1-5a28-41cf-aea7-e1fb050e8ee4/volumes"
Nov 03 18:17:33.493096 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:33.493054    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f79c02-5b47765c4-xmchm" podUID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.67:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:17:40.901008 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:40.900968    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"]
Nov 03 18:17:40.901580 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:40.901199    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82" podUID="d3f11d44-9763-4901-af47-ea7a5a7d0697" containerName="maas-api" containerID="cri-o://e9c82b23035e27e9854103f8e7fa4d2c8104fac00585ed52e180aba0e0cacf72" gracePeriod=30
Nov 03 18:17:41.145882 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.145839    6167 generic.go:356] "Generic (PLEG): container finished" podID="d3f11d44-9763-4901-af47-ea7a5a7d0697" containerID="e9c82b23035e27e9854103f8e7fa4d2c8104fac00585ed52e180aba0e0cacf72" exitCode=0
Nov 03 18:17:41.146052 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.145918    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82" event={"ID":"d3f11d44-9763-4901-af47-ea7a5a7d0697","Type":"ContainerDied","Data":"e9c82b23035e27e9854103f8e7fa4d2c8104fac00585ed52e180aba0e0cacf72"}
Nov 03 18:17:41.183281 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.183255    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:17:41.336152 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.336110    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls\") pod \"d3f11d44-9763-4901-af47-ea7a5a7d0697\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") "
Nov 03 18:17:41.336350 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.336180    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs\") pod \"d3f11d44-9763-4901-af47-ea7a5a7d0697\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") "
Nov 03 18:17:41.336350 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.336317    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca\") pod \"d3f11d44-9763-4901-af47-ea7a5a7d0697\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") "
Nov 03 18:17:41.336504 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.336438    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk\" (UniqueName: \"kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk\") pod \"d3f11d44-9763-4901-af47-ea7a5a7d0697\" (UID: \"d3f11d44-9763-4901-af47-ea7a5a7d0697\") "
Nov 03 18:17:41.336857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.336820    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca" pod "d3f11d44-9763-4901-af47-ea7a5a7d0697" (UID: "d3f11d44-9763-4901-af47-ea7a5a7d0697"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:41.336993 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.336926    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f11d44-9763-4901-af47-ea7a5a7d0697-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:41.339021 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.338992    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls" pod "d3f11d44-9763-4901-af47-ea7a5a7d0697" (UID: "d3f11d44-9763-4901-af47-ea7a5a7d0697"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:41.339186 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.339004    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk" pod "d3f11d44-9763-4901-af47-ea7a5a7d0697" (UID: "d3f11d44-9763-4901-af47-ea7a5a7d0697"). InnerVolumeSpecName "kube-api-access-hwmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:41.339549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.339521    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs" pod "d3f11d44-9763-4901-af47-ea7a5a7d0697" (UID: "d3f11d44-9763-4901-af47-ea7a5a7d0697"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:41.438366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.438282    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:41.438366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.438321    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f11d44-9763-4901-af47-ea7a5a7d0697-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:41.438366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:41.438337    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwmjk\" (UniqueName: \"kubernetes.io/projected/d3f11d44-9763-4901-af47-ea7a5a7d0697-kube-api-access-hwmjk\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:42.152543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:42.152512    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"
Nov 03 18:17:42.152543 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:42.152530    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82" event={"ID":"d3f11d44-9763-4901-af47-ea7a5a7d0697","Type":"ContainerDied","Data":"0fe9a688641b6b8bb467770ff63d40252f217cf95a0a1acb9282fe25549e189a"}
Nov 03 18:17:42.153086 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:42.152583    6167 scope.go:122] "RemoveContainer" containerID="e9c82b23035e27e9854103f8e7fa4d2c8104fac00585ed52e180aba0e0cacf72"
Nov 03 18:17:42.320171 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:42.320138    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"]
Nov 03 18:17:42.370609 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:42.370576    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1720c00c-6977b66858-gdw82"]
Nov 03 18:17:42.928188 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:42.928152    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f11d44-9763-4901-af47-ea7a5a7d0697" path="/var/lib/kubelet/pods/d3f11d44-9763-4901-af47-ea7a5a7d0697/volumes"
Nov 03 18:17:45.004163 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.004068    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" containerID="cri-o://125c2973d27f73b8e3bf8817e82b89d404c2befe2c908bb316b8e8f0319c208f" gracePeriod=45
Nov 03 18:17:45.175406 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.175368    6167 generic.go:356] "Generic (PLEG): container finished" podID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerID="125c2973d27f73b8e3bf8817e82b89d404c2befe2c908bb316b8e8f0319c208f" exitCode=0
Nov 03 18:17:45.175606 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.175443    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" event={"ID":"1541ddfd-21ad-420c-af7e-45604fa1f28d","Type":"ContainerDied","Data":"125c2973d27f73b8e3bf8817e82b89d404c2befe2c908bb316b8e8f0319c208f"}
Nov 03 18:17:45.288914 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.288882    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:17:45.382375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382336    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir\") pod \"1541ddfd-21ad-420c-af7e-45604fa1f28d\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") "
Nov 03 18:17:45.382574 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382385    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm\") pod \"1541ddfd-21ad-420c-af7e-45604fa1f28d\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") "
Nov 03 18:17:45.382574 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382457    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45\" (UniqueName: \"kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45\") pod \"1541ddfd-21ad-420c-af7e-45604fa1f28d\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") "
Nov 03 18:17:45.382574 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382507    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home\") pod \"1541ddfd-21ad-420c-af7e-45604fa1f28d\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") "
Nov 03 18:17:45.382574 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382533    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache\") pod \"1541ddfd-21ad-420c-af7e-45604fa1f28d\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") "
Nov 03 18:17:45.382850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382647    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir" pod "1541ddfd-21ad-420c-af7e-45604fa1f28d" (UID: "1541ddfd-21ad-420c-af7e-45604fa1f28d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.382850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.382756    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs\" (UniqueName: \"kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs\") pod \"1541ddfd-21ad-420c-af7e-45604fa1f28d\" (UID: \"1541ddfd-21ad-420c-af7e-45604fa1f28d\") "
Nov 03 18:17:45.383457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.383212    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.383457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.383407    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache" pod "1541ddfd-21ad-420c-af7e-45604fa1f28d" (UID: "1541ddfd-21ad-420c-af7e-45604fa1f28d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.383870 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.383826    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home" pod "1541ddfd-21ad-420c-af7e-45604fa1f28d" (UID: "1541ddfd-21ad-420c-af7e-45604fa1f28d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.386142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.385501    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs" pod "1541ddfd-21ad-420c-af7e-45604fa1f28d" (UID: "1541ddfd-21ad-420c-af7e-45604fa1f28d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:45.386344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.386317    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm" pod "1541ddfd-21ad-420c-af7e-45604fa1f28d" (UID: "1541ddfd-21ad-420c-af7e-45604fa1f28d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:45.386521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.386489    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45" pod "1541ddfd-21ad-420c-af7e-45604fa1f28d" (UID: "1541ddfd-21ad-420c-af7e-45604fa1f28d"). InnerVolumeSpecName "kube-api-access-wkq45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:45.483887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.483848    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1541ddfd-21ad-420c-af7e-45604fa1f28d-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.483887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.483878    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.483887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.483892    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkq45\" (UniqueName: \"kubernetes.io/projected/1541ddfd-21ad-420c-af7e-45604fa1f28d-kube-api-access-wkq45\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.484123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.483904    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.484123 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.483918    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1541ddfd-21ad-420c-af7e-45604fa1f28d-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:17:45.922340 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:45.922307    6167 scope.go:122] "RemoveContainer" containerID="125c2973d27f73b8e3bf8817e82b89d404c2befe2c908bb316b8e8f0319c208f"
Nov 03 18:17:46.182140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:46.182051    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" event={"ID":"1541ddfd-21ad-420c-af7e-45604fa1f28d","Type":"ContainerDied","Data":"1c0060d31a7624e73a79e48d5e2f0765d0658203d4d0969f2e64bc7e26f69fd6"}
Nov 03 18:17:46.182140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:46.182093    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"
Nov 03 18:17:46.208460 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:46.208424    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:17:46.415086 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:46.415051    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"]
Nov 03 18:17:46.419037 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:46.419011    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/premium-w5-2b4f46-kserve-6d477c649-k7n56"]
Nov 03 18:17:46.927900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:17:46.927866    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" path="/var/lib/kubelet/pods/1541ddfd-21ad-420c-af7e-45604fa1f28d/volumes"
Nov 03 18:17:54.521820 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:17:54.521778    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:18:00.699210 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:00.699170    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"]
Nov 03 18:18:00.699611 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:00.699385    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" containerID="cri-o://e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc" gracePeriod=30
Nov 03 18:18:01.022536 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:01.022459    6167 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:18:01.022536 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:01.022521    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:18:05.978246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:05.978188    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 18:18:06.129792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.129760    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.129997 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.129827    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.129997 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.129861    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.129997 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.129933    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.129997 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.129964    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.130222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130008    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.130222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130077    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.130222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130163    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.130222 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130182    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.130432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130242    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx\") pod \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\" (UID: \"251f15e8-9939-48fc-b99d-7ac8ab7cdedf\") "
Nov 03 18:18:06.130432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130380    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.130432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130326    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.130595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130463    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.130595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130487    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:06.130754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130705    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.130754 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130750    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.130955 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130765    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.130955 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130779    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.130955 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.130792    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.132119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.132096    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:06.132232 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.132182    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:06.132293 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.132281    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:06.132447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.132422    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx" pod "251f15e8-9939-48fc-b99d-7ac8ab7cdedf" (UID: "251f15e8-9939-48fc-b99d-7ac8ab7cdedf"). InnerVolumeSpecName "kube-api-access-c24hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:06.231976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.231892    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.231976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.231924    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24hx\" (UniqueName: \"kubernetes.io/projected/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-kube-api-access-c24hx\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.231976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.231939    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.231976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.231951    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251f15e8-9939-48fc-b99d-7ac8ab7cdedf-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:06.313902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.313861    6167 generic.go:356] "Generic (PLEG): container finished" podID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerID="e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc" exitCode=0
Nov 03 18:18:06.314077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.313900    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" event={"ID":"251f15e8-9939-48fc-b99d-7ac8ab7cdedf","Type":"ContainerDied","Data":"e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc"}
Nov 03 18:18:06.314077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.313932    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"
Nov 03 18:18:06.314077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.313953    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628" event={"ID":"251f15e8-9939-48fc-b99d-7ac8ab7cdedf","Type":"ContainerDied","Data":"8b58b9da1b9a8de354313ad2d316b8ff2e68cc0ab37940e0dbe2d9409a13c5d0"}
Nov 03 18:18:06.314077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.313982    6167 scope.go:122] "RemoveContainer" containerID="e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc"
Nov 03 18:18:06.324881 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.324804    6167 scope.go:122] "RemoveContainer" containerID="4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"
Nov 03 18:18:06.333540 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.333522    6167 scope.go:122] "RemoveContainer" containerID="e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc"
Nov 03 18:18:06.333888 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:18:06.333868    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc\": container with ID starting with e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc not found: ID does not exist" containerID="e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc"
Nov 03 18:18:06.333954 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.333904    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc"} err="failed to get container status \"e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc\": rpc error: code = NotFound desc = could not find container \"e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc\": container with ID starting with e959f1a9337ae9e77bb454d063cf1b8bba09c08a7446d464a3b9e3508e6f16dc not found: ID does not exist"
Nov 03 18:18:06.333954 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.333923    6167 scope.go:122] "RemoveContainer" containerID="4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"
Nov 03 18:18:06.334208 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:18:06.334180    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f\": container with ID starting with 4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f not found: ID does not exist" containerID="4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"
Nov 03 18:18:06.334280 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.334219    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f"} err="failed to get container status \"4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f\": rpc error: code = NotFound desc = could not find container \"4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f\": container with ID starting with 4a84b97c261eee924d22d36bb0e99d01a25b83d157f82fda3d744eff0a06f86f not found: ID does not exist"
Nov 03 18:18:06.480191 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.480153    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"]
Nov 03 18:18:06.484142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.484073    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-qv628"]
Nov 03 18:18:06.928141 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:06.928109    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" path="/var/lib/kubelet/pods/251f15e8-9939-48fc-b99d-7ac8ab7cdedf/volumes"
Nov 03 18:18:11.950021 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:11.949982    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:18:24.522095 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:18:24.522057    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:18:37.791791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:37.791713    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"]
Nov 03 18:18:37.792309 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:37.791941    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" containerID="cri-o://bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da" gracePeriod=30
Nov 03 18:18:40.457246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:40.457209    6167 patch_prober.go:29] interesting pod/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:18:40.457700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:40.457268    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:18:43.087831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.087801    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:18:43.247282 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247243    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247316    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247348    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247467 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247403    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247652 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247601    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247749 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247698    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.247821 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247745    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.247821 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247777    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247817    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247849    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.247913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.247900    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs\") pod \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\" (UID: \"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f\") "
Nov 03 18:18:43.248073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248039    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.248316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248285    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.248437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248377    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:43.248506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248467    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.248506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248490    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.248613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248505    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.248613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.248520    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.249654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.249631    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:43.249798 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.249654    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "kube-api-access-gn2mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:43.250024 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.250007    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:43.250167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.250144    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo" pod "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" (UID: "105b7f3f-5a8f-434b-aa34-8bc6d64ee01f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:43.353579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.353535    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.353579 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.353580    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.353860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.353596    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.353860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.353610    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.353860 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.353626    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2mf\" (UniqueName: \"kubernetes.io/projected/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f-kube-api-access-gn2mf\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:18:43.558141 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.558100    6167 generic.go:356] "Generic (PLEG): container finished" podID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerID="bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da" exitCode=0
Nov 03 18:18:43.558356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.558146    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"
Nov 03 18:18:43.558356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.558186    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" event={"ID":"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f","Type":"ContainerDied","Data":"bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da"}
Nov 03 18:18:43.558356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.558249    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f" event={"ID":"105b7f3f-5a8f-434b-aa34-8bc6d64ee01f","Type":"ContainerDied","Data":"298e8ac5167569d2609e7e9850dd2cc5d8597f482d1258dd6b4ebd95999758c0"}
Nov 03 18:18:43.558356 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.558279    6167 scope.go:122] "RemoveContainer" containerID="bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da"
Nov 03 18:18:43.573495 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.573468    6167 scope.go:122] "RemoveContainer" containerID="bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da"
Nov 03 18:18:43.573894 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:18:43.573864    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da\": container with ID starting with bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da not found: ID does not exist" containerID="bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da"
Nov 03 18:18:43.574004 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.573908    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da"} err="failed to get container status \"bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da\": rpc error: code = NotFound desc = could not find container \"bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da\": container with ID starting with bd3544b5e165b19399716642e601d76ba676637d90673b286dfa57b915f8b6da not found: ID does not exist"
Nov 03 18:18:43.727247 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.727148    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"]
Nov 03 18:18:43.777433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:43.777125    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1720c00c-openshift-default-5c55b8fbd5-6wg4f"]
Nov 03 18:18:44.927113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:18:44.927083    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" path="/var/lib/kubelet/pods/105b7f3f-5a8f-434b-aa34-8bc6d64ee01f/volumes"
Nov 03 18:18:54.521121 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:18:54.521044    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:19:24.521175 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:24.521135    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:19:31.564638 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:31.564595    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"]
Nov 03 18:19:31.610096 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:31.610060    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"]
Nov 03 18:19:32.884185 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:32.884152    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"]
Nov 03 18:19:32.884659 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:32.884398    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy" containerID="cri-o://34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e" gracePeriod=30
Nov 03 18:19:37.925279 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:37.925248    6167 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8c00df_c0fe_41ea_b486_88c3d0e1be36.slice/crio-34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 18:19:38.148419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.148331    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:19:38.289777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.289676    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.289777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.289739    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.289777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.289762    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.289784    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.289915    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.289960    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290049    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:38.290065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290066    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290343 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290102    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290343 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290136    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs\") pod \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\" (UID: \"cb8c00df-c0fe-41ea-b486-88c3d0e1be36\") "
Nov 03 18:19:38.290458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290368    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:38.290458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290395    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:38.290458 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290396    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:38.290623 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290584    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:38.290751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290732    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.290833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290757    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.290833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290771    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.290833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290787    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.290833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.290803    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.292153 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.292117    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "kube-api-access-4zsvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:38.292289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.292190    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:38.292289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.292199    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:38.292289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.292189    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo" pod "cb8c00df-c0fe-41ea-b486-88c3d0e1be36" (UID: "cb8c00df-c0fe-41ea-b486-88c3d0e1be36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:19:38.392375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.392336    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.392375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.392369    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.392375 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.392378    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.392624 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.392388    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsvx\" (UniqueName: \"kubernetes.io/projected/cb8c00df-c0fe-41ea-b486-88c3d0e1be36-kube-api-access-4zsvx\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.749131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.749094    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"]
Nov 03 18:19:38.957554 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.957516    6167 generic.go:356] "Generic (PLEG): container finished" podID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerID="34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e" exitCode=0
Nov 03 18:19:38.958116 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.957624    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"
Nov 03 18:19:38.958116 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.957632    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" event={"ID":"cb8c00df-c0fe-41ea-b486-88c3d0e1be36","Type":"ContainerDied","Data":"34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e"}
Nov 03 18:19:38.958116 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.957709    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw" event={"ID":"cb8c00df-c0fe-41ea-b486-88c3d0e1be36","Type":"ContainerDied","Data":"164900d110b3f4daae0701dfffa10e7cbaebecf91c6dfe6a31d0b12632d7f625"}
Nov 03 18:19:38.958116 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.957767    6167 scope.go:122] "RemoveContainer" containerID="34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e"
Nov 03 18:19:38.969130 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.969107    6167 scope.go:122] "RemoveContainer" containerID="34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e"
Nov 03 18:19:38.969426 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:38.969406    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e\": container with ID starting with 34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e not found: ID does not exist" containerID="34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e"
Nov 03 18:19:38.969513 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:38.969438    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e"} err="failed to get container status \"34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e\": rpc error: code = NotFound desc = could not find container \"34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e\": container with ID starting with 34d6dae53c096f3aed0fade2971453ce590287b8006b029a11a3ade63cd3062e not found: ID does not exist"
Nov 03 18:19:39.135916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:39.135876    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"]
Nov 03 18:19:39.185042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:39.185000    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-v77kw"]
Nov 03 18:19:40.927907 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:40.927872    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" path="/var/lib/kubelet/pods/cb8c00df-c0fe-41ea-b486-88c3d0e1be36/volumes"
Nov 03 18:19:43.749649 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:43.749589    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerName="payload-pre-processing" containerID="cri-o://494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd" gracePeriod=25
Nov 03 18:19:44.302603 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:44.302567    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:19:46.606511 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.606461    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main" containerID="cri-o://9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab" gracePeriod=45
Nov 03 18:19:46.635008 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.634954    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" containerID="cri-o://e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412" gracePeriod=45
Nov 03 18:19:46.763923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.763883    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused"
Nov 03 18:19:46.900518 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.900489    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:19:46.928960 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.928938    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:19:46.993900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.993867    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir\") pod \"df67e709-136e-428c-af59-abd8a1671d30\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") "
Nov 03 18:19:46.994067 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.993950    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache\") pod \"df67e709-136e-428c-af59-abd8a1671d30\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") "
Nov 03 18:19:46.994067 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994013    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624\" (UniqueName: \"kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624\") pod \"df67e709-136e-428c-af59-abd8a1671d30\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") "
Nov 03 18:19:46.994160 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994097    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home\") pod \"df67e709-136e-428c-af59-abd8a1671d30\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") "
Nov 03 18:19:46.994228 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994181    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir" pod "df67e709-136e-428c-af59-abd8a1671d30" (UID: "df67e709-136e-428c-af59-abd8a1671d30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:46.994228 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994215    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm\") pod \"df67e709-136e-428c-af59-abd8a1671d30\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") "
Nov 03 18:19:46.994337 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994257    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs\" (UniqueName: \"kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs\") pod \"df67e709-136e-428c-af59-abd8a1671d30\" (UID: \"df67e709-136e-428c-af59-abd8a1671d30\") "
Nov 03 18:19:46.994411 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994390    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home" pod "df67e709-136e-428c-af59-abd8a1671d30" (UID: "df67e709-136e-428c-af59-abd8a1671d30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:46.994655 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994628    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache" pod "df67e709-136e-428c-af59-abd8a1671d30" (UID: "df67e709-136e-428c-af59-abd8a1671d30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:46.994808 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994787    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:46.994902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994812    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:46.994902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.994826    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:46.996804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.996778    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624" pod "df67e709-136e-428c-af59-abd8a1671d30" (UID: "df67e709-136e-428c-af59-abd8a1671d30"). InnerVolumeSpecName "kube-api-access-q4624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:46.997103 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.997081    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs" pod "df67e709-136e-428c-af59-abd8a1671d30" (UID: "df67e709-136e-428c-af59-abd8a1671d30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:46.998836 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:46.998811    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm" pod "df67e709-136e-428c-af59-abd8a1671d30" (UID: "df67e709-136e-428c-af59-abd8a1671d30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:47.003763 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.003708    6167 generic.go:356] "Generic (PLEG): container finished" podID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerID="e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412" exitCode=0
Nov 03 18:19:47.003886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.003765    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" event={"ID":"552ff4e2-6eb5-45cd-8268-a3595f91dce1","Type":"ContainerDied","Data":"e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412"}
Nov 03 18:19:47.003886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.003791    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"
Nov 03 18:19:47.003886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.003809    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr" event={"ID":"552ff4e2-6eb5-45cd-8268-a3595f91dce1","Type":"ContainerDied","Data":"338755b788f4f90b5ed1bc620fef15c3a1e63bf4ef968430ea0c588eb9d95417"}
Nov 03 18:19:47.003886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.003828    6167 scope.go:122] "RemoveContainer" containerID="e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412"
Nov 03 18:19:47.005484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.005459    6167 generic.go:356] "Generic (PLEG): container finished" podID="df67e709-136e-428c-af59-abd8a1671d30" containerID="9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab" exitCode=0
Nov 03 18:19:47.005589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.005504    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"
Nov 03 18:19:47.005589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.005539    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" event={"ID":"df67e709-136e-428c-af59-abd8a1671d30","Type":"ContainerDied","Data":"9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab"}
Nov 03 18:19:47.005589 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.005583    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc" event={"ID":"df67e709-136e-428c-af59-abd8a1671d30","Type":"ContainerDied","Data":"9e90595ca591078b79d9cc9b37b54f189969035a1f6f871ed8aebe961f19e508"}
Nov 03 18:19:47.021594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.021567    6167 scope.go:122] "RemoveContainer" containerID="e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412"
Nov 03 18:19:47.022029 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:47.022000    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412\": container with ID starting with e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412 not found: ID does not exist" containerID="e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412"
Nov 03 18:19:47.022142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.022042    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412"} err="failed to get container status \"e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412\": rpc error: code = NotFound desc = could not find container \"e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412\": container with ID starting with e4532ed315788cfba5c4f0937559ea79bc90dc8d8a10f7ffb280d21a0afe3412 not found: ID does not exist"
Nov 03 18:19:47.022142 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.022067    6167 scope.go:122] "RemoveContainer" containerID="9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab"
Nov 03 18:19:47.031600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.031575    6167 scope.go:122] "RemoveContainer" containerID="9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab"
Nov 03 18:19:47.031979 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:47.031960    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab\": container with ID starting with 9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab not found: ID does not exist" containerID="9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab"
Nov 03 18:19:47.032091 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.031985    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab"} err="failed to get container status \"9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab\": rpc error: code = NotFound desc = could not find container \"9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab\": container with ID starting with 9b75d899e58d886fb7bf8f7380e315f99bf5093a772a56db1db46332712c22ab not found: ID does not exist"
Nov 03 18:19:47.095895 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.095856    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs\" (UniqueName: \"kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs\") pod \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") "
Nov 03 18:19:47.096140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.095925    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home\") pod \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") "
Nov 03 18:19:47.096140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.095968    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache\") pod \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") "
Nov 03 18:19:47.096140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096014    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc\" (UniqueName: \"kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc\") pod \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") "
Nov 03 18:19:47.096140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096045    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm\") pod \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") "
Nov 03 18:19:47.096140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096097    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir\") pod \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\" (UID: \"552ff4e2-6eb5-45cd-8268-a3595f91dce1\") "
Nov 03 18:19:47.096607 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096579    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache" pod "552ff4e2-6eb5-45cd-8268-a3595f91dce1" (UID: "552ff4e2-6eb5-45cd-8268-a3595f91dce1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:47.096855 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096610    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df67e709-136e-428c-af59-abd8a1671d30-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.096956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096872    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df67e709-136e-428c-af59-abd8a1671d30-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.096956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096891    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4624\" (UniqueName: \"kubernetes.io/projected/df67e709-136e-428c-af59-abd8a1671d30-kube-api-access-q4624\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.096956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096689    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home" pod "552ff4e2-6eb5-45cd-8268-a3595f91dce1" (UID: "552ff4e2-6eb5-45cd-8268-a3595f91dce1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:47.096956 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.096863    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir" pod "552ff4e2-6eb5-45cd-8268-a3595f91dce1" (UID: "552ff4e2-6eb5-45cd-8268-a3595f91dce1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:47.099201 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.099179    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs" pod "552ff4e2-6eb5-45cd-8268-a3595f91dce1" (UID: "552ff4e2-6eb5-45cd-8268-a3595f91dce1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:47.099304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.099226    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc" pod "552ff4e2-6eb5-45cd-8268-a3595f91dce1" (UID: "552ff4e2-6eb5-45cd-8268-a3595f91dce1"). InnerVolumeSpecName "kube-api-access-2mdjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:47.099351 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.099332    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm" pod "552ff4e2-6eb5-45cd-8268-a3595f91dce1" (UID: "552ff4e2-6eb5-45cd-8268-a3595f91dce1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:47.131598 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.131518    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"]
Nov 03 18:19:47.142365 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.142334    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-w1-ef9633-kserve-9db868c58-t8vcc"]
Nov 03 18:19:47.198871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.198800    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdjc\" (UniqueName: \"kubernetes.io/projected/552ff4e2-6eb5-45cd-8268-a3595f91dce1-kube-api-access-2mdjc\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.198871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.198839    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.198871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.198854    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.198871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.198867    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552ff4e2-6eb5-45cd-8268-a3595f91dce1-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.198871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.198879    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.199665 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.198896    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552ff4e2-6eb5-45cd-8268-a3595f91dce1-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:47.426013 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.425931    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"]
Nov 03 18:19:47.433072 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:47.433041    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/premium-w1-ef9633-kserve-757b57bfbc-f62wr"]
Nov 03 18:19:48.927187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:48.927157    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" path="/var/lib/kubelet/pods/552ff4e2-6eb5-45cd-8268-a3595f91dce1/volumes"
Nov 03 18:19:48.927569 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:48.927550    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df67e709-136e-428c-af59-abd8a1671d30" path="/var/lib/kubelet/pods/df67e709-136e-428c-af59-abd8a1671d30/volumes"
Nov 03 18:19:49.351359 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.351321    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"]
Nov 03 18:19:49.351575 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.351548    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" podUID="86a1f178-4833-4db8-95a3-d597ec75369a" containerName="discovery" containerID="cri-o://bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863" gracePeriod=30
Nov 03 18:19:49.613020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.612959    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:19:49.725262 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725231    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.725476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725285    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.725476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725371    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.725476 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725445    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.725594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725498    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.725594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725538    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.725715 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725693    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:49.725815 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.725796    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig\") pod \"86a1f178-4833-4db8-95a3-d597ec75369a\" (UID: \"86a1f178-4833-4db8-95a3-d597ec75369a\") "
Nov 03 18:19:49.726279 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.726250    6167 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-ca-configmap\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:49.727898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.727867    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:49.728029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.727876    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "kube-api-access-vhjk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:49.728029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.728012    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:19:49.728116 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.728094    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:49.728268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.728252    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:49.728357 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.728337    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig" pod "86a1f178-4833-4db8-95a3-d597ec75369a" (UID: "86a1f178-4833-4db8-95a3-d597ec75369a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:49.827244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.827200    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:49.827244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.827237    6167 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-kubeconfig\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:49.827472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.827254    6167 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-istio-csr-dns-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:49.827472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.827267    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjk8\" (UniqueName: \"kubernetes.io/projected/86a1f178-4833-4db8-95a3-d597ec75369a-kube-api-access-vhjk8\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:49.827472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.827280    6167 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a1f178-4833-4db8-95a3-d597ec75369a-local-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:49.827472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:49.827291    6167 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86a1f178-4833-4db8-95a3-d597ec75369a-cacerts\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:50.025891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.025795    6167 generic.go:356] "Generic (PLEG): container finished" podID="86a1f178-4833-4db8-95a3-d597ec75369a" containerID="bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863" exitCode=0
Nov 03 18:19:50.025891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.025855    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"
Nov 03 18:19:50.026451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.025878    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" event={"ID":"86a1f178-4833-4db8-95a3-d597ec75369a","Type":"ContainerDied","Data":"bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863"}
Nov 03 18:19:50.026451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.025947    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv" event={"ID":"86a1f178-4833-4db8-95a3-d597ec75369a","Type":"ContainerDied","Data":"ad20a9e500c8bcbe5626829a68dc7f68683bee5fedf5f9708ffa1bbc4cadc4ca"}
Nov 03 18:19:50.026451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.025968    6167 scope.go:122] "RemoveContainer" containerID="bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863"
Nov 03 18:19:50.036965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.036948    6167 scope.go:122] "RemoveContainer" containerID="bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863"
Nov 03 18:19:50.037217 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:50.037200    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863\": container with ID starting with bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863 not found: ID does not exist" containerID="bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863"
Nov 03 18:19:50.037271 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.037224    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863"} err="failed to get container status \"bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863\": rpc error: code = NotFound desc = could not find container \"bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863\": container with ID starting with bba37e63b501f6e3c9377250a051b7424f0fd9ab59b3958f8d236c12d3852863 not found: ID does not exist"
Nov 03 18:19:50.192027 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.191993    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"]
Nov 03 18:19:50.196095 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.196074    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5pnsv"]
Nov 03 18:19:50.928168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:50.928137    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a1f178-4833-4db8-95a3-d597ec75369a" path="/var/lib/kubelet/pods/86a1f178-4833-4db8-95a3-d597ec75369a/volumes"
Nov 03 18:19:54.302384 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:54.302344    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:19:54.520300 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:54.520266    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:19:57.106902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.106863    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"]
Nov 03 18:19:57.107385 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.107049    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" podUID="eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" containerName="maas-api" containerID="cri-o://169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3" gracePeriod=30
Nov 03 18:19:57.395619 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.395588    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 18:19:57.503594 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.503559    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca\") pod \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") "
Nov 03 18:19:57.503834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.503650    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs\") pod \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") "
Nov 03 18:19:57.503834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.503758    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx\") pod \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") "
Nov 03 18:19:57.503834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.503803    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\") pod \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\" (UID: \"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad\") "
Nov 03 18:19:57.504901 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.504868    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca" pod "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" (UID: "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:57.507215 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.507192    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs" pod "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" (UID: "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:57.510210 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.510166    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx" pod "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" (UID: "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad"). InnerVolumeSpecName "kube-api-access-lqfkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:57.538978 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.520899    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls" pod "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" (UID: "eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:57.605533 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.605488    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:57.605533 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.605535    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:57.605777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.605552    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-kube-api-access-lqfkx\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:57.605777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:57.605567    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:19:58.078383 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.078343    6167 generic.go:356] "Generic (PLEG): container finished" podID="eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" containerID="169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3" exitCode=0
Nov 03 18:19:58.078570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.078393    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" event={"ID":"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad","Type":"ContainerDied","Data":"169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3"}
Nov 03 18:19:58.078570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.078430    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"
Nov 03 18:19:58.078570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.078449    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr" event={"ID":"eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad","Type":"ContainerDied","Data":"cf431d14a3c8adca859029b39606cc5e71fed870fff3a4ad1439d9e194d734a8"}
Nov 03 18:19:58.078570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.078479    6167 scope.go:122] "RemoveContainer" containerID="169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3"
Nov 03 18:19:58.092422 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.092306    6167 scope.go:122] "RemoveContainer" containerID="169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3"
Nov 03 18:19:58.092686 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:19:58.092647    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3\": container with ID starting with 169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3 not found: ID does not exist" containerID="169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3"
Nov 03 18:19:58.092904 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.092682    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3"} err="failed to get container status \"169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3\": rpc error: code = NotFound desc = could not find container \"169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3\": container with ID starting with 169828b129a13c66bc1cb23ac6e8cc6464a801a90425641e168ae94d8dbc3de3 not found: ID does not exist"
Nov 03 18:19:58.246822 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.246772    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"]
Nov 03 18:19:58.297372 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.297336    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ef9633-d6c77d69d-69rcr"]
Nov 03 18:19:58.695397 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.695370    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:19:58.927608 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:19:58.927577    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" path="/var/lib/kubelet/pods/eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad/volumes"
Nov 03 18:20:04.017062 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.017002    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 18:20:04.069429 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.069391    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr\" (UniqueName: \"kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr\") pod \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") "
Nov 03 18:20:04.069601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.069445    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume\") pod \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\" (UID: \"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd\") "
Nov 03 18:20:04.069958 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.069901    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume" pod "0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" (UID: "0987cbc1-cbe0-4eda-b554-82f2dc76e4cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:04.071980 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.071949    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr" pod "0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" (UID: "0987cbc1-cbe0-4eda-b554-82f2dc76e4cd"). InnerVolumeSpecName "kube-api-access-ltskr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:04.114115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.114079    6167 generic.go:356] "Generic (PLEG): container finished" podID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerID="494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd" exitCode=0
Nov 03 18:20:04.114277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.114126    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"
Nov 03 18:20:04.114277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.114156    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" event={"ID":"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd","Type":"ContainerDied","Data":"494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd"}
Nov 03 18:20:04.114277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.114225    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg" event={"ID":"0987cbc1-cbe0-4eda-b554-82f2dc76e4cd","Type":"ContainerDied","Data":"abc47ad9c24400baf565512b24d10921bdd061e6b6957ee9e96073be6a514bcd"}
Nov 03 18:20:04.114277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.114248    6167 scope.go:122] "RemoveContainer" containerID="494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd"
Nov 03 18:20:04.124586 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.124561    6167 scope.go:122] "RemoveContainer" containerID="494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd"
Nov 03 18:20:04.124885 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:20:04.124864    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd\": container with ID starting with 494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd not found: ID does not exist" containerID="494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd"
Nov 03 18:20:04.124983 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.124894    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd"} err="failed to get container status \"494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd\": rpc error: code = NotFound desc = could not find container \"494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd\": container with ID starting with 494f42e18d4a5c757277c170b99307ddb1963f0d9d55b13c3bade32dfd080afd not found: ID does not exist"
Nov 03 18:20:04.171290 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.171252    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltskr\" (UniqueName: \"kubernetes.io/projected/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-kube-api-access-ltskr\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:04.171290 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.171288    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:04.247777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.247742    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"]
Nov 03 18:20:04.251354 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.251318    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ef9633-577fc76cd7-728cg"]
Nov 03 18:20:04.928846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:04.928797    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" path="/var/lib/kubelet/pods/0987cbc1-cbe0-4eda-b554-82f2dc76e4cd/volumes"
Nov 03 18:20:06.972760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:06.972632    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"]
Nov 03 18:20:06.973139 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:06.972898    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy" containerID="cri-o://a13b7cf80164a11afd3e2985f94f98cff3c1825d5bd2b3915631290f06bcf2d1" gracePeriod=30
Nov 03 18:20:12.169220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.168986    6167 generic.go:356] "Generic (PLEG): container finished" podID="e87b6388-baa5-4920-910a-87b7a8589191" containerID="a13b7cf80164a11afd3e2985f94f98cff3c1825d5bd2b3915631290f06bcf2d1" exitCode=0
Nov 03 18:20:12.169971 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.169070    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" event={"ID":"e87b6388-baa5-4920-910a-87b7a8589191","Type":"ContainerDied","Data":"a13b7cf80164a11afd3e2985f94f98cff3c1825d5bd2b3915631290f06bcf2d1"}
Nov 03 18:20:12.235692 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.235658    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:20:12.354381 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354350    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354399    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354478    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354510    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354538    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354569    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354613 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354613    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354664    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.354965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354675    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354777    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert\") pod \"e87b6388-baa5-4920-910a-87b7a8589191\" (UID: \"e87b6388-baa5-4920-910a-87b7a8589191\") "
Nov 03 18:20:12.354965 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.354905    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.355194 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355008    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.355194 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355105    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.355306 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355288    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:12.355393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355371    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.355434 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355399    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.355434 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355417    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.355501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.355431    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.356893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.356863    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:12.356893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.356877    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "kube-api-access-nxrs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:12.357180 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.357152    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:12.357180 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.357154    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo" pod "e87b6388-baa5-4920-910a-87b7a8589191" (UID: "e87b6388-baa5-4920-910a-87b7a8589191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:20:12.456731 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.456681    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.456875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.456741    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxrs6\" (UniqueName: \"kubernetes.io/projected/e87b6388-baa5-4920-910a-87b7a8589191-kube-api-access-nxrs6\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.456875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.456763    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e87b6388-baa5-4920-910a-87b7a8589191-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.456875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.456778    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e87b6388-baa5-4920-910a-87b7a8589191-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:12.456875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:12.456792    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e87b6388-baa5-4920-910a-87b7a8589191-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:13.176916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:13.176882    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg" event={"ID":"e87b6388-baa5-4920-910a-87b7a8589191","Type":"ContainerDied","Data":"9bd4bbe9f0efdc6a89f5d3c8be9f4d6812771b18268774dbbd88ad9a4029920b"}
Nov 03 18:20:13.176916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:13.176900    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"
Nov 03 18:20:13.177366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:13.176942    6167 scope.go:122] "RemoveContainer" containerID="a13b7cf80164a11afd3e2985f94f98cff3c1825d5bd2b3915631290f06bcf2d1"
Nov 03 18:20:13.343879 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:13.343846    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"]
Nov 03 18:20:13.393932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:13.393851    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-hvvlg"]
Nov 03 18:20:14.284497 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.284461    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:20:14.532728 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.532683    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-775tp"]
Nov 03 18:20:14.533151 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533136    6167 cpu_manager.go:414] "containerMap: removing container" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533153    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533165    6167 cpu_manager.go:414] "containerMap: removing container" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533171    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533180    6167 cpu_manager.go:414] "containerMap: removing container" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533185    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533197    6167 cpu_manager.go:414] "containerMap: removing container" podUID="86a1f178-4833-4db8-95a3-d597ec75369a" containerName="discovery"
Nov 03 18:20:14.533213 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533203    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a1f178-4833-4db8-95a3-d597ec75369a" containerName="discovery"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533226    6167 cpu_manager.go:414] "containerMap: removing container" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerName="payload-pre-processing"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533234    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerName="payload-pre-processing"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533248    6167 cpu_manager.go:414] "containerMap: removing container" podUID="eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533256    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533270    6167 cpu_manager.go:414] "containerMap: removing container" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533276    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533289    6167 cpu_manager.go:414] "containerMap: removing container" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533295    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533304    6167 cpu_manager.go:414] "containerMap: removing container" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533310    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533320    6167 cpu_manager.go:414] "containerMap: removing container" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533327    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533342    6167 cpu_manager.go:414] "containerMap: removing container" podUID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533349    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533365    6167 cpu_manager.go:414] "containerMap: removing container" podUID="11f7da85-fbe3-47ec-8bb5-4e138ac94e86" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533371    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f7da85-fbe3-47ec-8bb5-4e138ac94e86" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533384    6167 cpu_manager.go:414] "containerMap: removing container" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533391    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533411    6167 cpu_manager.go:414] "containerMap: removing container" podUID="d3f11d44-9763-4901-af47-ea7a5a7d0697" containerName="maas-api"
Nov 03 18:20:14.533432 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533417    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f11d44-9763-4901-af47-ea7a5a7d0697" containerName="maas-api"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533552    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87b6388-baa5-4920-910a-87b7a8589191" containerName="istio-proxy"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533568    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="1541ddfd-21ad-420c-af7e-45604fa1f28d" containerName="main"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533577    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="0214d7cf-6f58-4889-99d8-f98e8553a400" containerName="main"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533590    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab055d1-5a28-41cf-aea7-e1fb050e8ee4" containerName="maas-api"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533601    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f7da85-fbe3-47ec-8bb5-4e138ac94e86" containerName="maas-api"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533609    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d3bbfc-f466-49d3-b876-910aa65efd6a" containerName="istio-proxy"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533621    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="df67e709-136e-428c-af59-abd8a1671d30" containerName="main"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533633    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533643    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f11d44-9763-4901-af47-ea7a5a7d0697" containerName="maas-api"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533654    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="552ff4e2-6eb5-45cd-8268-a3595f91dce1" containerName="main"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533665    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="105b7f3f-5a8f-434b-aa34-8bc6d64ee01f" containerName="istio-proxy"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533674    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533685    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a1f178-4833-4db8-95a3-d597ec75369a" containerName="discovery"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533699    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="0987cbc1-cbe0-4eda-b554-82f2dc76e4cd" containerName="payload-pre-processing"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533709    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae8c4b2-dfe0-4c5f-8618-aa2612fa2dad" containerName="maas-api"
Nov 03 18:20:14.534031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.533736    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8c00df-c0fe-41ea-b486-88c3d0e1be36" containerName="istio-proxy"
Nov 03 18:20:14.538347 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.538300    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.591461 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.591433    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 18:20:14.646950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.642052    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 18:20:14.646950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.642402    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 18:20:14.646950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.642433    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kcmdt\""
Nov 03 18:20:14.646950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.645930    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-775tp"]
Nov 03 18:20:14.683083 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.683045    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp5s\" (UniqueName: \"kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.683402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.683377    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.683542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.683505    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.784601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.784549    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp5s\" (UniqueName: \"kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.784984 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.784618    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.784984 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.784697    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.788150 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.788112    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.789048 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.788978    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.848980 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.848942    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp5s\" (UniqueName: \"kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s\") pod \"maas-controller-5c8b7c4cfc-775tp\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.852187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.851893    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:14.954520 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:14.950194    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87b6388-baa5-4920-910a-87b7a8589191" path="/var/lib/kubelet/pods/e87b6388-baa5-4920-910a-87b7a8589191/volumes"
Nov 03 18:20:15.250923 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:20:15.250885    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92747fba_cf15_4a16_98b3_1dc6016354c7.slice/crio-1974368f5f6ec5e3104d77b50f48df40c75bc6ac42c936c093d0543fec2ead60 WatchSource:0}: Error finding container 1974368f5f6ec5e3104d77b50f48df40c75bc6ac42c936c093d0543fec2ead60: Status 404 returned error can't find the container with id 1974368f5f6ec5e3104d77b50f48df40c75bc6ac42c936c093d0543fec2ead60
Nov 03 18:20:15.251322 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:15.251292    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-775tp"]
Nov 03 18:20:15.252636 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:15.252614    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:20:16.211981 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:16.211931    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp" event={"ID":"92747fba-cf15-4a16-98b3-1dc6016354c7","Type":"ContainerStarted","Data":"1974368f5f6ec5e3104d77b50f48df40c75bc6ac42c936c093d0543fec2ead60"}
Nov 03 18:20:19.235745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:19.235681    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp" event={"ID":"92747fba-cf15-4a16-98b3-1dc6016354c7","Type":"ContainerStarted","Data":"4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e"}
Nov 03 18:20:19.236236 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:19.235811    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:19.354372 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:19.354313    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp" podStartSLOduration=2.022327612 podStartE2EDuration="5.354291993s" totalImagesPullingTime="3.331964381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:19.353940942 +0000 UTC m=+2074.584266015" watchObservedRunningTime="2026-11-03 18:20:19.354291993 +0000 UTC m=+2074.584617066"
Nov 03 18:20:24.521014 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:20:24.520931    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:20:30.244134 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:30.244105    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:45.343981 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:45.343954    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:20:45.350864 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:45.350837    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:20:52.054103 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.054061    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-775tp"]
Nov 03 18:20:52.054703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.054259    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp" podUID="92747fba-cf15-4a16-98b3-1dc6016354c7" containerName="manager" containerID="cri-o://4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e" gracePeriod=30
Nov 03 18:20:52.316563 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.316539    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:52.382088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.382054    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert\") pod \"92747fba-cf15-4a16-98b3-1dc6016354c7\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") "
Nov 03 18:20:52.382268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.382113    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s\" (UniqueName: \"kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s\") pod \"92747fba-cf15-4a16-98b3-1dc6016354c7\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") "
Nov 03 18:20:52.382268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.382189    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs\") pod \"92747fba-cf15-4a16-98b3-1dc6016354c7\" (UID: \"92747fba-cf15-4a16-98b3-1dc6016354c7\") "
Nov 03 18:20:52.384212 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.384175    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s" pod "92747fba-cf15-4a16-98b3-1dc6016354c7" (UID: "92747fba-cf15-4a16-98b3-1dc6016354c7"). InnerVolumeSpecName "kube-api-access-lhp5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:52.384353 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.384254    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs" pod "92747fba-cf15-4a16-98b3-1dc6016354c7" (UID: "92747fba-cf15-4a16-98b3-1dc6016354c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:52.384353 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.384277    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert" pod "92747fba-cf15-4a16-98b3-1dc6016354c7" (UID: "92747fba-cf15-4a16-98b3-1dc6016354c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:52.436669 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.436635    6167 generic.go:356] "Generic (PLEG): container finished" podID="92747fba-cf15-4a16-98b3-1dc6016354c7" containerID="4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e" exitCode=0
Nov 03 18:20:52.436869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.436691    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp"
Nov 03 18:20:52.436869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.436703    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp" event={"ID":"92747fba-cf15-4a16-98b3-1dc6016354c7","Type":"ContainerDied","Data":"4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e"}
Nov 03 18:20:52.436869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.436788    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-775tp" event={"ID":"92747fba-cf15-4a16-98b3-1dc6016354c7","Type":"ContainerDied","Data":"1974368f5f6ec5e3104d77b50f48df40c75bc6ac42c936c093d0543fec2ead60"}
Nov 03 18:20:52.436869 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.436820    6167 scope.go:122] "RemoveContainer" containerID="4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e"
Nov 03 18:20:52.463615 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.463594    6167 scope.go:122] "RemoveContainer" containerID="4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e"
Nov 03 18:20:52.467568 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:20:52.467541    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e\": container with ID starting with 4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e not found: ID does not exist" containerID="4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e"
Nov 03 18:20:52.467686 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.467580    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e"} err="failed to get container status \"4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e\": rpc error: code = NotFound desc = could not find container \"4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e\": container with ID starting with 4d5176a4aa6edb851f9a463d54e3af6cb2dd58384f8cad831e75be5bd3eedc5e not found: ID does not exist"
Nov 03 18:20:52.482949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.482923    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp5s\" (UniqueName: \"kubernetes.io/projected/92747fba-cf15-4a16-98b3-1dc6016354c7-kube-api-access-lhp5s\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:52.482949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.482949    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:52.483125 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.482960    6167 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92747fba-cf15-4a16-98b3-1dc6016354c7-webhook-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:20:52.602520 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.602440    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-775tp"]
Nov 03 18:20:52.607444 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.607416    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-775tp"]
Nov 03 18:20:52.932789 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:52.932696    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92747fba-cf15-4a16-98b3-1dc6016354c7" path="/var/lib/kubelet/pods/92747fba-cf15-4a16-98b3-1dc6016354c7/volumes"
Nov 03 18:20:54.024877 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.024841    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-vkfzr"]
Nov 03 18:20:54.025316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.025302    6167 cpu_manager.go:414] "containerMap: removing container" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy"
Nov 03 18:20:54.025364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.025317    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251f15e8-9939-48fc-b99d-7ac8ab7cdedf" containerName="istio-proxy"
Nov 03 18:20:54.025364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.025338    6167 cpu_manager.go:414] "containerMap: removing container" podUID="92747fba-cf15-4a16-98b3-1dc6016354c7" containerName="manager"
Nov 03 18:20:54.025364 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.025343    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92747fba-cf15-4a16-98b3-1dc6016354c7" containerName="manager"
Nov 03 18:20:54.025478 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.025425    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="92747fba-cf15-4a16-98b3-1dc6016354c7" containerName="manager"
Nov 03 18:20:54.030378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.030360    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.083739 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.083685    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 18:20:54.099096 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.099063    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f13c90-2d9c-427a-b3e5-db90108c2284-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.099251 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.099212    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91f13c90-2d9c-427a-b3e5-db90108c2284-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.099308 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.099259    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzf5k\" (UniqueName: \"kubernetes.io/projected/91f13c90-2d9c-427a-b3e5-db90108c2284-kube-api-access-hzf5k\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.134121 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.134078    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 18:20:54.134121 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.134104    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 18:20:54.134344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.134169    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kcmdt\""
Nov 03 18:20:54.136126 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.136096    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-vkfzr"]
Nov 03 18:20:54.200237 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.200193    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91f13c90-2d9c-427a-b3e5-db90108c2284-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.200424 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.200250    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzf5k\" (UniqueName: \"kubernetes.io/projected/91f13c90-2d9c-427a-b3e5-db90108c2284-kube-api-access-hzf5k\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.200424 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.200291    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f13c90-2d9c-427a-b3e5-db90108c2284-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.202866 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.202837    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91f13c90-2d9c-427a-b3e5-db90108c2284-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.202995 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.202866    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f13c90-2d9c-427a-b3e5-db90108c2284-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.261804 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.261770    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzf5k\" (UniqueName: \"kubernetes.io/projected/91f13c90-2d9c-427a-b3e5-db90108c2284-kube-api-access-hzf5k\") pod \"maas-controller-5c8b7c4cfc-vkfzr\" (UID: \"91f13c90-2d9c-427a-b3e5-db90108c2284\") " pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.343942 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.343896    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:54.520942 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:20:54.520894    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:20:54.712830 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:20:54.712797    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f13c90_2d9c_427a_b3e5_db90108c2284.slice/crio-e8547fa7d7c8c64138c3a5547a9dd6387d75c8f5a7bf2457c60337cba17ac62c WatchSource:0}: Error finding container e8547fa7d7c8c64138c3a5547a9dd6387d75c8f5a7bf2457c60337cba17ac62c: Status 404 returned error can't find the container with id e8547fa7d7c8c64138c3a5547a9dd6387d75c8f5a7bf2457c60337cba17ac62c
Nov 03 18:20:54.714842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:54.714814    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-vkfzr"]
Nov 03 18:20:55.455929 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:55.455897    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr" event={"ID":"91f13c90-2d9c-427a-b3e5-db90108c2284","Type":"ContainerStarted","Data":"e8547fa7d7c8c64138c3a5547a9dd6387d75c8f5a7bf2457c60337cba17ac62c"}
Nov 03 18:20:56.464021 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:56.463989    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr" event={"ID":"91f13c90-2d9c-427a-b3e5-db90108c2284","Type":"ContainerStarted","Data":"9e5ff0e4a53648bfdd9ba3157fa32b8a719d4284dacf0fbf6ba4a15efa4fb95c"}
Nov 03 18:20:56.464415 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:56.464117    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:20:56.585010 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:20:56.584954    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr" podStartSLOduration=2.786360009 podStartE2EDuration="3.584940592s" totalImagesPullingTime="798.580583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:56.583306581 +0000 UTC m=+2111.813631653" watchObservedRunningTime="2026-11-03 18:20:56.584940592 +0000 UTC m=+2111.815265663"
Nov 03 18:21:07.472522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:21:07.472492    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c8b7c4cfc-vkfzr"
Nov 03 18:21:24.521061 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:21:24.521022    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:21:37.485834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:21:37.485797    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:21:54.520813 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:21:54.520708    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:21:58.887048 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:21:58.887014    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:22:06.464935 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:22:06.464871    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:22:06.465591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:22:06.465560    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:22:08.679966 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:22:08.679934    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:22:24.520538 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:22:24.520500    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:22:30.251249 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:22:30.251215    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:22:47.148274 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:22:47.148235    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:22:54.520770 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:22:54.520699    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:23:18.260522 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:23:18.260484    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:23:24.520172 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:23:24.520085    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:23:28.981085 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:23:28.981050    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:23:41.664099 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:23:41.664063    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:23:54.521305 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:23:54.521263    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:24:03.991836 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:24:03.991796    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:24:13.583031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:24:13.582999    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:24:24.521267 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:24:24.521229    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:24:32.292248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:24:32.292209    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:24:44.883370 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:24:44.883335    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:24:54.520927 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:24:54.520837    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:25:04.863278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:04.863236    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:25:07.200298 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:07.199872    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:25:07.200298 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:07.199941    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:25:24.064823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:24.064781    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:25:24.602740 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:25:24.602601    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:25:37.093778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:37.093744    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:25:45.379542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:45.379509    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:25:45.388012 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:25:45.387987    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:25:54.521486 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:25:54.521449    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:26:10.154400 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:26:10.154364    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:26:24.520945 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:26:24.520851    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:26:31.986331 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:26:31.986285    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:26:42.681426 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:26:42.681389    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:26:54.520697 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:26:54.520655    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:27:15.485359 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.485326    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"]
Nov 03 18:27:15.489042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.489024    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.498930 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.498903    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpfvp\" (UniqueName: \"kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp\") pod \"payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.499069 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.498961    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.541978 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.541931    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-58e97a\""
Nov 03 18:27:15.544760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.544711    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"]
Nov 03 18:27:15.600030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.599991    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpfvp\" (UniqueName: \"kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp\") pod \"payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.600224 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.600116    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.600950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.600928    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.635578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.635543    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"]
Nov 03 18:27:15.639471 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.639450    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.662814 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.662790    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpfvp\" (UniqueName: \"kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp\") pod \"payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.692517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.692485    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b24de7e2-dockercfg-hvk6q\""
Nov 03 18:27:15.695456 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.695425    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"]
Nov 03 18:27:15.701271 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.701242    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.701391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.701291    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jml7x\" (UniqueName: \"kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x\") pod \"payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.802401 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.802313    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:15.802570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.802544    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.802683 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.802589    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jml7x\" (UniqueName: \"kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x\") pod \"payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.803303 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.803282    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.865195 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.865123    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jml7x\" (UniqueName: \"kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x\") pod \"payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:15.955060 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:15.955021    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:16.173466 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:27:16.173050    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf889baf4_d51e_4506_9a65_9d607966b1c0.slice/crio-961cc15675712fb63b39782e1c7cce44b2e6a420b571dc2891bcaa75e5041717 WatchSource:0}: Error finding container 961cc15675712fb63b39782e1c7cce44b2e6a420b571dc2891bcaa75e5041717: Status 404 returned error can't find the container with id 961cc15675712fb63b39782e1c7cce44b2e6a420b571dc2891bcaa75e5041717
Nov 03 18:27:16.175751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.174694    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"]
Nov 03 18:27:16.322441 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:27:16.322409    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2916f72_8f33_44b1_b4e7_f12ecf3b1b2f.slice/crio-0cedd5cd9cc8ecdc90eeeedc55753d3a91468c20667ee092d9d4606974f1444e WatchSource:0}: Error finding container 0cedd5cd9cc8ecdc90eeeedc55753d3a91468c20667ee092d9d4606974f1444e: Status 404 returned error can't find the container with id 0cedd5cd9cc8ecdc90eeeedc55753d3a91468c20667ee092d9d4606974f1444e
Nov 03 18:27:16.324840 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.324817    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"]
Nov 03 18:27:16.656600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.656558    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" event={"ID":"f889baf4-d51e-4506-9a65-9d607966b1c0","Type":"ContainerStarted","Data":"aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d"}
Nov 03 18:27:16.657211 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.656621    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" event={"ID":"f889baf4-d51e-4506-9a65-9d607966b1c0","Type":"ContainerStarted","Data":"961cc15675712fb63b39782e1c7cce44b2e6a420b571dc2891bcaa75e5041717"}
Nov 03 18:27:16.657211 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.656755    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:16.658302 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.658278    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" event={"ID":"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f","Type":"ContainerStarted","Data":"1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164"}
Nov 03 18:27:16.658421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.658319    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:16.658421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.658335    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" event={"ID":"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f","Type":"ContainerStarted","Data":"0cedd5cd9cc8ecdc90eeeedc55753d3a91468c20667ee092d9d4606974f1444e"}
Nov 03 18:27:16.774115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.774063    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" podStartSLOduration=1.774048879 podStartE2EDuration="1.774048879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:16.774005552 +0000 UTC m=+2492.004330623" watchObservedRunningTime="2026-11-03 18:27:16.774048879 +0000 UTC m=+2492.004373952"
Nov 03 18:27:16.891609 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:16.891544    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" podStartSLOduration=1.891524488 podStartE2EDuration="1.891524488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:16.891014576 +0000 UTC m=+2492.121339649" watchObservedRunningTime="2026-11-03 18:27:16.891524488 +0000 UTC m=+2492.121849561"
Nov 03 18:27:18.109776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.109743    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"]
Nov 03 18:27:18.114599 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.114573    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.125518 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.125490    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xq2v\" (UniqueName: \"kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.125677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.125654    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.125746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.125702    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.125853 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.125841    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.167317 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.167293    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-58e97a-serving-cert\""
Nov 03 18:27:18.219850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.219818    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"]
Nov 03 18:27:18.226534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.226507    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.226629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.226556    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.226629 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.226618    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.226772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.226639    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xq2v\" (UniqueName: \"kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.226839 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:27:18.226766    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-serving-cert: secret "maas-api-e2e-worker-single-main-58e97a-serving-cert" not found
Nov 03 18:27:18.226897 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:27:18.226840    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls podName:15df8b31-cfe3-45c7-9f2c-7711b25a4fae nodeName:}" failed. No retries permitted until 2026-11-03 18:27:18.726819523 +0000 UTC m=+2493.957144597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls") pod "maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" (UID: "15df8b31-cfe3-45c7-9f2c-7711b25a4fae") : secret "maas-api-e2e-worker-single-main-58e97a-serving-cert" not found
Nov 03 18:27:18.227268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.227246    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.229392 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.229364    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.288582 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.288546    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xq2v\" (UniqueName: \"kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.731035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.731003    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:18.733316 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:18.733292    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:19.028564 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:19.028472    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:19.403087 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:27:19.403057    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15df8b31_cfe3_45c7_9f2c_7711b25a4fae.slice/crio-e37c0acb7e682157efe8846c3ed05e186ee25d869b86c30f957f7ae39488a5e0 WatchSource:0}: Error finding container e37c0acb7e682157efe8846c3ed05e186ee25d869b86c30f957f7ae39488a5e0: Status 404 returned error can't find the container with id e37c0acb7e682157efe8846c3ed05e186ee25d869b86c30f957f7ae39488a5e0
Nov 03 18:27:19.403898 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:19.403814    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"]
Nov 03 18:27:19.404756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:19.404739    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:27:19.677890 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:19.677788    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" event={"ID":"15df8b31-cfe3-45c7-9f2c-7711b25a4fae","Type":"ContainerStarted","Data":"e37c0acb7e682157efe8846c3ed05e186ee25d869b86c30f957f7ae39488a5e0"}
Nov 03 18:27:20.685538 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:20.685504    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" event={"ID":"15df8b31-cfe3-45c7-9f2c-7711b25a4fae","Type":"ContainerStarted","Data":"3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d"}
Nov 03 18:27:20.686057 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:20.685618    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:20.804374 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:20.804272    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" podStartSLOduration=1.771827826 podStartE2EDuration="2.804257372s" totalImagesPullingTime="1.032429546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:20.803681965 +0000 UTC m=+2496.034007037" watchObservedRunningTime="2026-11-03 18:27:20.804257372 +0000 UTC m=+2496.034582443"
Nov 03 18:27:24.521025 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:27:24.520984    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:27:26.255252 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:26.255218    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:27:26.697727 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:26.697697    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:27:27.667582 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:27.667554    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:27:27.668012 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:27.667627    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:27:44.576223 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:27:44.576182    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:27:54.521540 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:27:54.521458    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:28:10.669849 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:10.669807    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:28:24.521287 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:28:24.521244    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:28:39.975531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:39.975490    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:28:48.714593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:48.714560    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"]
Nov 03 18:28:48.820115 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:48.820075    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"]
Nov 03 18:28:53.715886 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:53.715838    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerName="payload-processing" containerID="cri-o://1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164" gracePeriod=25
Nov 03 18:28:53.821656 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:53.821595    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerName="payload-pre-processing" containerID="cri-o://aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d" gracePeriod=25
Nov 03 18:28:54.522143 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:28:54.522028    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:28:55.192173 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:55.192140    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"]
Nov 03 18:28:55.192567 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:55.192331    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" podUID="15df8b31-cfe3-45c7-9f2c-7711b25a4fae" containerName="maas-api" containerID="cri-o://3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d" gracePeriod=30
Nov 03 18:28:56.459131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.459095    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:28:56.528084 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.527989    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs\") pod \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") "
Nov 03 18:28:56.528250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.528172    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v\" (UniqueName: \"kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v\") pod \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") "
Nov 03 18:28:56.528250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.528245    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca\") pod \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") "
Nov 03 18:28:56.528371 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.528273    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\") pod \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\" (UID: \"15df8b31-cfe3-45c7-9f2c-7711b25a4fae\") "
Nov 03 18:28:56.528602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.528572    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca" pod "15df8b31-cfe3-45c7-9f2c-7711b25a4fae" (UID: "15df8b31-cfe3-45c7-9f2c-7711b25a4fae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:28:56.530218 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.530182    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs" pod "15df8b31-cfe3-45c7-9f2c-7711b25a4fae" (UID: "15df8b31-cfe3-45c7-9f2c-7711b25a4fae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:28:56.530433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.530406    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls" pod "15df8b31-cfe3-45c7-9f2c-7711b25a4fae" (UID: "15df8b31-cfe3-45c7-9f2c-7711b25a4fae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:28:56.530433 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.530418    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v" pod "15df8b31-cfe3-45c7-9f2c-7711b25a4fae" (UID: "15df8b31-cfe3-45c7-9f2c-7711b25a4fae"). InnerVolumeSpecName "kube-api-access-6xq2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:28:56.629311 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.629276    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:56.629311 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.629308    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xq2v\" (UniqueName: \"kubernetes.io/projected/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-kube-api-access-6xq2v\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:56.629519 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.629320    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:56.629519 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:56.629329    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15df8b31-cfe3-45c7-9f2c-7711b25a4fae-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:28:57.233646 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.233558    6167 generic.go:356] "Generic (PLEG): container finished" podID="15df8b31-cfe3-45c7-9f2c-7711b25a4fae" containerID="3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d" exitCode=0
Nov 03 18:28:57.233646 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.233612    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"
Nov 03 18:28:57.233908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.233636    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" event={"ID":"15df8b31-cfe3-45c7-9f2c-7711b25a4fae","Type":"ContainerDied","Data":"3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d"}
Nov 03 18:28:57.233908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.233694    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq" event={"ID":"15df8b31-cfe3-45c7-9f2c-7711b25a4fae","Type":"ContainerDied","Data":"e37c0acb7e682157efe8846c3ed05e186ee25d869b86c30f957f7ae39488a5e0"}
Nov 03 18:28:57.233908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.233742    6167 scope.go:122] "RemoveContainer" containerID="3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d"
Nov 03 18:28:57.245293 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.245262    6167 scope.go:122] "RemoveContainer" containerID="3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d"
Nov 03 18:28:57.245832 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:28:57.245800    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d\": container with ID starting with 3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d not found: ID does not exist" containerID="3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d"
Nov 03 18:28:57.245948 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.245839    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d"} err="failed to get container status \"3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d\": rpc error: code = NotFound desc = could not find container \"3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d\": container with ID starting with 3f2a11847995e528ec0bdfbd6208f099bfe90486413f870349666fe14f3b160d not found: ID does not exist"
Nov 03 18:28:57.399597 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.399566    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"]
Nov 03 18:28:57.450088 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.450050    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-58e97a-574f8cd697-ncvmq"]
Nov 03 18:28:57.665976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.665938    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:28:57.665976 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:57.665959    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:28:58.927588 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:28:58.927552    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15df8b31-cfe3-45c7-9f2c-7711b25a4fae" path="/var/lib/kubelet/pods/15df8b31-cfe3-45c7-9f2c-7711b25a4fae/volumes"
Nov 03 18:29:00.078027 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:00.077985    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:29:07.665694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:07.665650    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:29:07.665694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:07.665674    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:29:13.979270 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:13.979203    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:29:14.016246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.015563    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume\") pod \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") "
Nov 03 18:29:14.016246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.015740    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x\" (UniqueName: \"kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x\") pod \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\" (UID: \"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f\") "
Nov 03 18:29:14.016246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.016183    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume" pod "c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" (UID: "c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:29:14.018474 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.018444    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x" pod "c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" (UID: "c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f"). InnerVolumeSpecName "kube-api-access-jml7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:14.055687 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.055666    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:29:14.117184 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.117156    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp\" (UniqueName: \"kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp\") pod \"f889baf4-d51e-4506-9a65-9d607966b1c0\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") "
Nov 03 18:29:14.117362 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.117221    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume\") pod \"f889baf4-d51e-4506-9a65-9d607966b1c0\" (UID: \"f889baf4-d51e-4506-9a65-9d607966b1c0\") "
Nov 03 18:29:14.117618 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.117586    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume" pod "f889baf4-d51e-4506-9a65-9d607966b1c0" (UID: "f889baf4-d51e-4506-9a65-9d607966b1c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:29:14.117756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.117700    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f889baf4-d51e-4506-9a65-9d607966b1c0-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:14.117756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.117737    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:14.117756 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.117755    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jml7x\" (UniqueName: \"kubernetes.io/projected/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f-kube-api-access-jml7x\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:14.119378 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.119350    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp" pod "f889baf4-d51e-4506-9a65-9d607966b1c0" (UID: "f889baf4-d51e-4506-9a65-9d607966b1c0"). InnerVolumeSpecName "kube-api-access-jpfvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:14.219278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.219243    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpfvp\" (UniqueName: \"kubernetes.io/projected/f889baf4-d51e-4506-9a65-9d607966b1c0-kube-api-access-jpfvp\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:14.327029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.326990    6167 generic.go:356] "Generic (PLEG): container finished" podID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerID="aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d" exitCode=0
Nov 03 18:29:14.327220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.327032    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"
Nov 03 18:29:14.327220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.327071    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" event={"ID":"f889baf4-d51e-4506-9a65-9d607966b1c0","Type":"ContainerDied","Data":"aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d"}
Nov 03 18:29:14.327220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.327133    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs" event={"ID":"f889baf4-d51e-4506-9a65-9d607966b1c0","Type":"ContainerDied","Data":"961cc15675712fb63b39782e1c7cce44b2e6a420b571dc2891bcaa75e5041717"}
Nov 03 18:29:14.327220 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.327167    6167 scope.go:122] "RemoveContainer" containerID="aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d"
Nov 03 18:29:14.328498 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.328477    6167 generic.go:356] "Generic (PLEG): container finished" podID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerID="1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164" exitCode=0
Nov 03 18:29:14.328591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.328529    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"
Nov 03 18:29:14.328591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.328531    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" event={"ID":"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f","Type":"ContainerDied","Data":"1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164"}
Nov 03 18:29:14.328697 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.328616    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs" event={"ID":"c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f","Type":"ContainerDied","Data":"0cedd5cd9cc8ecdc90eeeedc55753d3a91468c20667ee092d9d4606974f1444e"}
Nov 03 18:29:14.336937 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.336913    6167 scope.go:122] "RemoveContainer" containerID="aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d"
Nov 03 18:29:14.337215 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:29:14.337179    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d\": container with ID starting with aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d not found: ID does not exist" containerID="aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d"
Nov 03 18:29:14.337302 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.337227    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d"} err="failed to get container status \"aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d\": rpc error: code = NotFound desc = could not find container \"aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d\": container with ID starting with aad08ae47eb7f8d2408c37b9865f9543e4c64c60ba3c22d168d9911c5afac88d not found: ID does not exist"
Nov 03 18:29:14.337302 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.337250    6167 scope.go:122] "RemoveContainer" containerID="1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164"
Nov 03 18:29:14.345592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.345572    6167 scope.go:122] "RemoveContainer" containerID="1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164"
Nov 03 18:29:14.345874 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:29:14.345857    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164\": container with ID starting with 1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164 not found: ID does not exist" containerID="1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164"
Nov 03 18:29:14.345928 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.345891    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164"} err="failed to get container status \"1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164\": rpc error: code = NotFound desc = could not find container \"1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164\": container with ID starting with 1944e81897bee9d3d59dc4df103ce67ece052de876091017c76bffa5b15dd164 not found: ID does not exist"
Nov 03 18:29:14.493602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.493568    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"]
Nov 03 18:29:14.499301 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.499275    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-58e97a-5bcfd5bfbsgnbs"]
Nov 03 18:29:14.609705 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.609641    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"]
Nov 03 18:29:14.616677 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.616652    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-58e97a-7c85dwbdfs"]
Nov 03 18:29:14.926733 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.926644    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" path="/var/lib/kubelet/pods/c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f/volumes"
Nov 03 18:29:14.927050 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:14.927035    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" path="/var/lib/kubelet/pods/f889baf4-d51e-4506-9a65-9d607966b1c0/volumes"
Nov 03 18:29:24.520663 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:29:24.520583    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:29:27.818735 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:27.818684    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"]
Nov 03 18:29:27.819517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:27.818908    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh" podUID="34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" containerName="manager" containerID="cri-o://f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f" gracePeriod=10
Nov 03 18:29:28.092560 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.092532    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 18:29:28.163745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.163690    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume\") pod \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") "
Nov 03 18:29:28.163745 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.163741    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt\" (UniqueName: \"kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt\") pod \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\" (UID: \"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd\") "
Nov 03 18:29:28.164082 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.164061    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume" pod "34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" (UID: "34af34ed-2b37-4c4a-a3df-7d5ed334d4cd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:29:28.165816 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.165793    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt" pod "34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" (UID: "34af34ed-2b37-4c4a-a3df-7d5ed334d4cd"). InnerVolumeSpecName "kube-api-access-77mlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:29:28.264896 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.264861    6167 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-extensions-socket-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:28.264896 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.264892    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mlt\" (UniqueName: \"kubernetes.io/projected/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd-kube-api-access-77mlt\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:29:28.404706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.404623    6167 generic.go:356] "Generic (PLEG): container finished" podID="34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" containerID="f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f" exitCode=0
Nov 03 18:29:28.404706 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.404665    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"
Nov 03 18:29:28.404916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.404708    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh" event={"ID":"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd","Type":"ContainerDied","Data":"f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f"}
Nov 03 18:29:28.404916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.404823    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh" event={"ID":"34af34ed-2b37-4c4a-a3df-7d5ed334d4cd","Type":"ContainerDied","Data":"79d0e5f351d80c74a4f9b5b07bb51e6845f5f7cdd5dd750eabbf510e8ced6877"}
Nov 03 18:29:28.404916 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.404847    6167 scope.go:122] "RemoveContainer" containerID="f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f"
Nov 03 18:29:28.418385 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.418363    6167 scope.go:122] "RemoveContainer" containerID="f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f"
Nov 03 18:29:28.418669 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:29:28.418648    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f\": container with ID starting with f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f not found: ID does not exist" containerID="f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f"
Nov 03 18:29:28.418766 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.418677    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f"} err="failed to get container status \"f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f\": rpc error: code = NotFound desc = could not find container \"f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f\": container with ID starting with f082c89f93f7b434463a993ccc52dbc336e9facd02da1504c14a229a07adb31f not found: ID does not exist"
Nov 03 18:29:28.571176 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.571146    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"]
Nov 03 18:29:28.575926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.575904    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qhgfh"]
Nov 03 18:29:28.926646 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:29:28.926602    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" path="/var/lib/kubelet/pods/34af34ed-2b37-4c4a-a3df-7d5ed334d4cd/volumes"
Nov 03 18:29:54.520229 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:29:54.520190    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:30:24.521571 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:30:24.521527    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:30:31.491990 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.491955    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"]
Nov 03 18:30:31.492399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492385    6167 cpu_manager.go:414] "containerMap: removing container" podUID="15df8b31-cfe3-45c7-9f2c-7711b25a4fae" containerName="maas-api"
Nov 03 18:30:31.492399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492396    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15df8b31-cfe3-45c7-9f2c-7711b25a4fae" containerName="maas-api"
Nov 03 18:30:31.492473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492419    6167 cpu_manager.go:414] "containerMap: removing container" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerName="payload-pre-processing"
Nov 03 18:30:31.492473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492425    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerName="payload-pre-processing"
Nov 03 18:30:31.492473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492434    6167 cpu_manager.go:414] "containerMap: removing container" podUID="34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" containerName="manager"
Nov 03 18:30:31.492473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492439    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" containerName="manager"
Nov 03 18:30:31.492473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492458    6167 cpu_manager.go:414] "containerMap: removing container" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerName="payload-processing"
Nov 03 18:30:31.492473 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492462    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerName="payload-processing"
Nov 03 18:30:31.492654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492535    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="15df8b31-cfe3-45c7-9f2c-7711b25a4fae" containerName="maas-api"
Nov 03 18:30:31.492654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492545    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="f889baf4-d51e-4506-9a65-9d607966b1c0" containerName="payload-pre-processing"
Nov 03 18:30:31.492654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492551    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="34af34ed-2b37-4c4a-a3df-7d5ed334d4cd" containerName="manager"
Nov 03 18:30:31.492654 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.492560    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2916f72-8f33-44b1-b4e7-f12ecf3b1b2f" containerName="payload-processing"
Nov 03 18:30:31.495830 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.495811    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.545506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.545466    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"]
Nov 03 18:30:31.594076 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.594041    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vjv2n\""
Nov 03 18:30:31.597568 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.597532    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8pk\" (UniqueName: \"kubernetes.io/projected/849ddfdc-3b2b-4c8d-852f-75849b3d3ef8-kube-api-access-7t8pk\") pod \"kuadrant-operator-controller-manager-678d87766f-lqw2c\" (UID: \"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.597694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.597590    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/849ddfdc-3b2b-4c8d-852f-75849b3d3ef8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lqw2c\" (UID: \"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.698684 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.698645    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/849ddfdc-3b2b-4c8d-852f-75849b3d3ef8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lqw2c\" (UID: \"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.698908 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.698824    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8pk\" (UniqueName: \"kubernetes.io/projected/849ddfdc-3b2b-4c8d-852f-75849b3d3ef8-kube-api-access-7t8pk\") pod \"kuadrant-operator-controller-manager-678d87766f-lqw2c\" (UID: \"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.699146 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.699124    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/849ddfdc-3b2b-4c8d-852f-75849b3d3ef8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lqw2c\" (UID: \"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.762102 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.762035    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8pk\" (UniqueName: \"kubernetes.io/projected/849ddfdc-3b2b-4c8d-852f-75849b3d3ef8-kube-api-access-7t8pk\") pod \"kuadrant-operator-controller-manager-678d87766f-lqw2c\" (UID: \"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:31.812031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:31.811997    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:32.277830 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:30:32.277785    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849ddfdc_3b2b_4c8d_852f_75849b3d3ef8.slice/crio-9f7ac3a41e926c14cbfba3ac6f2647ae26c80de59ebf8c077b5315259bad92dc WatchSource:0}: Error finding container 9f7ac3a41e926c14cbfba3ac6f2647ae26c80de59ebf8c077b5315259bad92dc: Status 404 returned error can't find the container with id 9f7ac3a41e926c14cbfba3ac6f2647ae26c80de59ebf8c077b5315259bad92dc
Nov 03 18:30:32.280518 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:32.280487    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"]
Nov 03 18:30:32.762292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:32.762250    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c" event={"ID":"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8","Type":"ContainerStarted","Data":"07582eeea6f252fb2d4c03089090abab1bac3bab712429b3d0609dad82375f81"}
Nov 03 18:30:32.762671 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:32.762378    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:32.762671 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:32.762433    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c" event={"ID":"849ddfdc-3b2b-4c8d-852f-75849b3d3ef8","Type":"ContainerStarted","Data":"9f7ac3a41e926c14cbfba3ac6f2647ae26c80de59ebf8c077b5315259bad92dc"}
Nov 03 18:30:32.886344 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:32.886294    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c" podStartSLOduration=1.886281184 podStartE2EDuration="1.886281184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:32.884320771 +0000 UTC m=+2688.114645843" watchObservedRunningTime="2026-11-03 18:30:32.886281184 +0000 UTC m=+2688.116606259"
Nov 03 18:30:43.770506 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:43.770476    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lqw2c"
Nov 03 18:30:45.415542 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:45.415509    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:30:45.427969 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:45.427950    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:30:54.275129 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:30:54.275047    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:30:54.522521 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:30:54.522482    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:31:11.669119 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:11.669084    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:31:21.260421 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:21.260389    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:31:24.520837 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:31:24.520798    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:31:40.855558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:40.855516    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:31:49.105360 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.105326    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"]
Nov 03 18:31:49.109530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.109501    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.164857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.164827    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"]
Nov 03 18:31:49.214216 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.214186    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0d7aec\""
Nov 03 18:31:49.214422 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.214249    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7428cd65-dockercfg-t4w8m\""
Nov 03 18:31:49.221689 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.221659    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.221859 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.221799    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6t8d\" (UniqueName: \"kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d\") pod \"payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.274111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.274073    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"]
Nov 03 18:31:49.278008 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.277977    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.322601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.322573    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6t8d\" (UniqueName: \"kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d\") pod \"payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.322757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.322687    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.323399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.323376    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.332842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.332817    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"]
Nov 03 18:31:49.385140 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.385071    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6t8d\" (UniqueName: \"kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d\") pod \"payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.423402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.423365    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:49.423711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.423689    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lsq\" (UniqueName: \"kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq\") pod \"payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.423858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.423831    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.525521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.525473    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lsq\" (UniqueName: \"kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq\") pod \"payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.525694 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.525592    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.526319 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.526300    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.590446 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.590416    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lsq\" (UniqueName: \"kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq\") pod \"payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.591190 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.591170    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:49.790401 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:31:49.790372    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2856aa_00c0_45d8_b455_bf78132eb33e.slice/crio-a44a70cff486578682509e3c5f5cd93022450630454ddae4f229ac03aebf6adb WatchSource:0}: Error finding container a44a70cff486578682509e3c5f5cd93022450630454ddae4f229ac03aebf6adb: Status 404 returned error can't find the container with id a44a70cff486578682509e3c5f5cd93022450630454ddae4f229ac03aebf6adb
Nov 03 18:31:49.793571 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.793545    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"]
Nov 03 18:31:49.966709 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:31:49.966667    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe43a31_90eb_4915_b944_733e2708f2e7.slice/crio-3395bd05dc55f6d344a10da9ba5f53ef4566296e098d42990137c4edefc56943 WatchSource:0}: Error finding container 3395bd05dc55f6d344a10da9ba5f53ef4566296e098d42990137c4edefc56943: Status 404 returned error can't find the container with id 3395bd05dc55f6d344a10da9ba5f53ef4566296e098d42990137c4edefc56943
Nov 03 18:31:49.969699 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:49.969676    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"]
Nov 03 18:31:50.198517 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.198480    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" event={"ID":"ffe43a31-90eb-4915-b944-733e2708f2e7","Type":"ContainerStarted","Data":"93b7d796acde53a668dc8b33e7dfe5f727438d92e897edcfb5098739a407d994"}
Nov 03 18:31:50.198987 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.198544    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:31:50.198987 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.198568    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" event={"ID":"ffe43a31-90eb-4915-b944-733e2708f2e7","Type":"ContainerStarted","Data":"3395bd05dc55f6d344a10da9ba5f53ef4566296e098d42990137c4edefc56943"}
Nov 03 18:31:50.200182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.200153    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" event={"ID":"7a2856aa-00c0-45d8-b455-bf78132eb33e","Type":"ContainerStarted","Data":"31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1"}
Nov 03 18:31:50.200347 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.200193    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" event={"ID":"7a2856aa-00c0-45d8-b455-bf78132eb33e","Type":"ContainerStarted","Data":"a44a70cff486578682509e3c5f5cd93022450630454ddae4f229ac03aebf6adb"}
Nov 03 18:31:50.200347 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.200215    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:31:50.317152 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.317108    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" podStartSLOduration=1.317096453 podStartE2EDuration="1.317096453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:50.316066296 +0000 UTC m=+2765.546391370" watchObservedRunningTime="2026-11-03 18:31:50.317096453 +0000 UTC m=+2765.547421522"
Nov 03 18:31:50.435436 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:50.435380    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" podStartSLOduration=1.435367418 podStartE2EDuration="1.435367418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:50.434812514 +0000 UTC m=+2765.665137587" watchObservedRunningTime="2026-11-03 18:31:50.435367418 +0000 UTC m=+2765.665692487"
Nov 03 18:31:51.507262 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.507220    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"]
Nov 03 18:31:51.511250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.511224    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.543329 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.543289    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.543493 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.543378    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.543534 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.543490    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxb5\" (UniqueName: \"kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.543566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.543539    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.565393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.565354    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0d7aec-serving-cert\""
Nov 03 18:31:51.618777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.618744    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"]
Nov 03 18:31:51.644442 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.644389    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxb5\" (UniqueName: \"kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.644442 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.644446    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.644702 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.644501    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.644702 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.644542    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.645162 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:31:51.645138    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-serving-cert: secret "maas-api-e2e-worker-single-main-0d7aec-serving-cert" not found
Nov 03 18:31:51.645309 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:31:51.645213    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls podName:357dfddb-eaea-4fdf-a09a-f10792998f6e nodeName:}" failed. No retries permitted until 2026-11-03 18:31:52.145193363 +0000 UTC m=+2767.375518427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls") pod "maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" (UID: "357dfddb-eaea-4fdf-a09a-f10792998f6e") : secret "maas-api-e2e-worker-single-main-0d7aec-serving-cert" not found
Nov 03 18:31:51.645428 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.645401    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.653118 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.653085    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:51.707835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:51.707805    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxb5\" (UniqueName: \"kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:52.149276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:52.149230    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:52.151656 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:52.151624    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:52.425422 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:52.425328    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:52.801932 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:31:52.801894    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357dfddb_eaea_4fdf_a09a_f10792998f6e.slice/crio-41fc422c2eaee514fea270218b3812ba9bfcc97a27a1f863297bf482e4ceb842 WatchSource:0}: Error finding container 41fc422c2eaee514fea270218b3812ba9bfcc97a27a1f863297bf482e4ceb842: Status 404 returned error can't find the container with id 41fc422c2eaee514fea270218b3812ba9bfcc97a27a1f863297bf482e4ceb842
Nov 03 18:31:52.804949 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:52.804904    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"]
Nov 03 18:31:53.218602 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:53.218565    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" event={"ID":"357dfddb-eaea-4fdf-a09a-f10792998f6e","Type":"ContainerStarted","Data":"41fc422c2eaee514fea270218b3812ba9bfcc97a27a1f863297bf482e4ceb842"}
Nov 03 18:31:54.226693 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:54.226650    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" event={"ID":"357dfddb-eaea-4fdf-a09a-f10792998f6e","Type":"ContainerStarted","Data":"3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208"}
Nov 03 18:31:54.227126 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:54.226782    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:31:54.347815 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:54.347758    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" podStartSLOduration=2.539722295 podStartE2EDuration="3.347738748s" totalImagesPullingTime="808.016453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:54.346865841 +0000 UTC m=+2769.577190921" watchObservedRunningTime="2026-11-03 18:31:54.347738748 +0000 UTC m=+2769.578063821"
Nov 03 18:31:54.535471 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:31:54.535385    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:31:59.674583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:31:59.674550    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:32:00.237604 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:32:00.237570    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:32:01.209339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:32:01.209308    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:32:01.209811 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:32:01.209376    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:32:18.582105 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:32:18.582072    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:32:24.522796 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:32:24.522700    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:32:44.364061 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:32:44.364022    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:32:54.521123 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:32:54.521075    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:33:12.243362 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:12.243322    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:33:22.569042 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:22.569002    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"]
Nov 03 18:33:22.676451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:22.676411    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"]
Nov 03 18:33:24.521015 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:33:24.520978    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:33:27.569483 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:27.569423    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerName="payload-processing" containerID="cri-o://31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1" gracePeriod=25
Nov 03 18:33:27.677423 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:27.677380    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerName="payload-pre-processing" containerID="cri-o://93b7d796acde53a668dc8b33e7dfe5f727438d92e897edcfb5098739a407d994" gracePeriod=25
Nov 03 18:33:28.895566 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:28.895521    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"]
Nov 03 18:33:28.896100 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:28.895803    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" podUID="357dfddb-eaea-4fdf-a09a-f10792998f6e" containerName="maas-api" containerID="cri-o://3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208" gracePeriod=30
Nov 03 18:33:30.148692 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.148665    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:33:30.209442 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.209411    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\") pod \"357dfddb-eaea-4fdf-a09a-f10792998f6e\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") "
Nov 03 18:33:30.209595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.209511    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5\" (UniqueName: \"kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5\") pod \"357dfddb-eaea-4fdf-a09a-f10792998f6e\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") "
Nov 03 18:33:30.209595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.209542    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca\") pod \"357dfddb-eaea-4fdf-a09a-f10792998f6e\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") "
Nov 03 18:33:30.209595 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.209562    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs\") pod \"357dfddb-eaea-4fdf-a09a-f10792998f6e\" (UID: \"357dfddb-eaea-4fdf-a09a-f10792998f6e\") "
Nov 03 18:33:30.209967 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.209940    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca" pod "357dfddb-eaea-4fdf-a09a-f10792998f6e" (UID: "357dfddb-eaea-4fdf-a09a-f10792998f6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:30.211572 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.211552    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls" pod "357dfddb-eaea-4fdf-a09a-f10792998f6e" (UID: "357dfddb-eaea-4fdf-a09a-f10792998f6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:30.211648 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.211580    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs" pod "357dfddb-eaea-4fdf-a09a-f10792998f6e" (UID: "357dfddb-eaea-4fdf-a09a-f10792998f6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:33:30.211702 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.211687    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5" pod "357dfddb-eaea-4fdf-a09a-f10792998f6e" (UID: "357dfddb-eaea-4fdf-a09a-f10792998f6e"). InnerVolumeSpecName "kube-api-access-pfxb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:30.310558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.310483    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.310558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.310513    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxb5\" (UniqueName: \"kubernetes.io/projected/357dfddb-eaea-4fdf-a09a-f10792998f6e-kube-api-access-pfxb5\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.310558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.310523    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357dfddb-eaea-4fdf-a09a-f10792998f6e-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.310558 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.310532    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/357dfddb-eaea-4fdf-a09a-f10792998f6e-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:30.770707 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.770667    6167 generic.go:356] "Generic (PLEG): container finished" podID="357dfddb-eaea-4fdf-a09a-f10792998f6e" containerID="3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208" exitCode=0
Nov 03 18:33:30.770912 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.770756    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"
Nov 03 18:33:30.770912 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.770760    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" event={"ID":"357dfddb-eaea-4fdf-a09a-f10792998f6e","Type":"ContainerDied","Data":"3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208"}
Nov 03 18:33:30.770912 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.770842    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr" event={"ID":"357dfddb-eaea-4fdf-a09a-f10792998f6e","Type":"ContainerDied","Data":"41fc422c2eaee514fea270218b3812ba9bfcc97a27a1f863297bf482e4ceb842"}
Nov 03 18:33:30.770912 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.770865    6167 scope.go:122] "RemoveContainer" containerID="3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208"
Nov 03 18:33:30.781644 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.781626    6167 scope.go:122] "RemoveContainer" containerID="3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208"
Nov 03 18:33:30.781947 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:33:30.781927    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208\": container with ID starting with 3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208 not found: ID does not exist" containerID="3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208"
Nov 03 18:33:30.782003 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.781955    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208"} err="failed to get container status \"3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208\": rpc error: code = NotFound desc = could not find container \"3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208\": container with ID starting with 3055000229a39834aef8ee51dd4a437dbdf4e86c9783a19e5ec1965d2b328208 not found: ID does not exist"
Nov 03 18:33:30.939110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.939073    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"]
Nov 03 18:33:30.990381 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:30.990352    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0d7aec-7f6fb4c594-tlfrr"]
Nov 03 18:33:31.207379 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:31.207341    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:33:31.207777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:31.207341    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:33:32.927544 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:32.927502    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357dfddb-eaea-4fdf-a09a-f10792998f6e" path="/var/lib/kubelet/pods/357dfddb-eaea-4fdf-a09a-f10792998f6e/volumes"
Nov 03 18:33:33.667821 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:33.667784    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:33:41.207597 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:41.207562    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:33:41.208130 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:41.207561    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:33:44.776777 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.776742    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"]
Nov 03 18:33:44.777224 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.777211    6167 cpu_manager.go:414] "containerMap: removing container" podUID="357dfddb-eaea-4fdf-a09a-f10792998f6e" containerName="maas-api"
Nov 03 18:33:44.777268 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.777225    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357dfddb-eaea-4fdf-a09a-f10792998f6e" containerName="maas-api"
Nov 03 18:33:44.777331 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.777322    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="357dfddb-eaea-4fdf-a09a-f10792998f6e" containerName="maas-api"
Nov 03 18:33:44.780531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.780514    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.857664 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.857625    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.857857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.857750    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.857857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.857789    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtz6\" (UniqueName: \"kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.857857 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.857844    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.857996 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.857971    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.858113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.858093    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.885484 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.885461    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a3447aea-kserve-self-signed-certs\""
Nov 03 18:33:44.885601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.885485    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"]
Nov 03 18:33:44.959398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959364    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959398 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959399    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959649 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959454    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959649 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959491    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959649 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959523    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959649 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959553    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtz6\" (UniqueName: \"kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959794    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959913 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959875    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.959989 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.959962    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:44.961673 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:44.961645    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:45.012343 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.012316    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a3447aea-kserve-self-signed-certs\""
Nov 03 18:33:45.022171 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.022143    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:45.026084 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.026062    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtz6\" (UniqueName: \"kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6\") pod \"e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") " pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:45.105832 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.105808    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:45.482678 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:33:45.482649    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbdf801e_9d86_466c_8fa1_328870a44baf.slice/crio-5e766fdded0e2c6c2095fe8048f797210b340bb2cfd45771e4208eb0541c98f3 WatchSource:0}: Error finding container 5e766fdded0e2c6c2095fe8048f797210b340bb2cfd45771e4208eb0541c98f3: Status 404 returned error can't find the container with id 5e766fdded0e2c6c2095fe8048f797210b340bb2cfd45771e4208eb0541c98f3
Nov 03 18:33:45.484652 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.484569    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"]
Nov 03 18:33:45.857420 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.857380    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58" event={"ID":"cbdf801e-9d86-466c-8fa1-328870a44baf","Type":"ContainerStarted","Data":"4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c"}
Nov 03 18:33:45.857420 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.857427    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58" event={"ID":"cbdf801e-9d86-466c-8fa1-328870a44baf","Type":"ContainerStarted","Data":"5e766fdded0e2c6c2095fe8048f797210b340bb2cfd45771e4208eb0541c98f3"}
Nov 03 18:33:45.977571 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:45.977480    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58" podStartSLOduration=1.977466725 podStartE2EDuration="1.977466725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:45.9758654 +0000 UTC m=+2881.206190474" watchObservedRunningTime="2026-11-03 18:33:45.977466725 +0000 UTC m=+2881.207791787"
Nov 03 18:33:46.105935 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:46.105903    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:47.840239 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.840172    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:33:47.869510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.869270    6167 generic.go:356] "Generic (PLEG): container finished" podID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerID="31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1" exitCode=0
Nov 03 18:33:47.869510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.869318    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"
Nov 03 18:33:47.869510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.869323    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" event={"ID":"7a2856aa-00c0-45d8-b455-bf78132eb33e","Type":"ContainerDied","Data":"31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1"}
Nov 03 18:33:47.869510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.869432    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw" event={"ID":"7a2856aa-00c0-45d8-b455-bf78132eb33e","Type":"ContainerDied","Data":"a44a70cff486578682509e3c5f5cd93022450630454ddae4f229ac03aebf6adb"}
Nov 03 18:33:47.869510 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.869468    6167 scope.go:122] "RemoveContainer" containerID="31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1"
Nov 03 18:33:47.871514 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.871487    6167 generic.go:356] "Generic (PLEG): container finished" podID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerID="93b7d796acde53a668dc8b33e7dfe5f727438d92e897edcfb5098739a407d994" exitCode=0
Nov 03 18:33:47.871850 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.871827    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" event={"ID":"ffe43a31-90eb-4915-b944-733e2708f2e7","Type":"ContainerDied","Data":"93b7d796acde53a668dc8b33e7dfe5f727438d92e897edcfb5098739a407d994"}
Nov 03 18:33:47.879960 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.879933    6167 scope.go:122] "RemoveContainer" containerID="31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1"
Nov 03 18:33:47.880311 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:33:47.880278    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1\": container with ID starting with 31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1 not found: ID does not exist" containerID="31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1"
Nov 03 18:33:47.880394 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.880322    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1"} err="failed to get container status \"31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1\": rpc error: code = NotFound desc = could not find container \"31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1\": container with ID starting with 31ba54a20fa3d83062eb361871254c622fd563ec5cedd2222ad84603513835e1 not found: ID does not exist"
Nov 03 18:33:47.887751 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.887697    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d\" (UniqueName: \"kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d\") pod \"7a2856aa-00c0-45d8-b455-bf78132eb33e\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") "
Nov 03 18:33:47.887858 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.887817    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume\") pod \"7a2856aa-00c0-45d8-b455-bf78132eb33e\" (UID: \"7a2856aa-00c0-45d8-b455-bf78132eb33e\") "
Nov 03 18:33:47.888138 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.888114    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume" pod "7a2856aa-00c0-45d8-b455-bf78132eb33e" (UID: "7a2856aa-00c0-45d8-b455-bf78132eb33e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:47.888280 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.888259    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2856aa-00c0-45d8-b455-bf78132eb33e-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:47.890936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.890910    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d" pod "7a2856aa-00c0-45d8-b455-bf78132eb33e" (UID: "7a2856aa-00c0-45d8-b455-bf78132eb33e"). InnerVolumeSpecName "kube-api-access-s6t8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:47.916156 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.916136    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:33:47.989294 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.989267    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq\" (UniqueName: \"kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq\") pod \"ffe43a31-90eb-4915-b944-733e2708f2e7\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") "
Nov 03 18:33:47.989451 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.989409    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume\") pod \"ffe43a31-90eb-4915-b944-733e2708f2e7\" (UID: \"ffe43a31-90eb-4915-b944-733e2708f2e7\") "
Nov 03 18:33:47.989672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.989641    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6t8d\" (UniqueName: \"kubernetes.io/projected/7a2856aa-00c0-45d8-b455-bf78132eb33e-kube-api-access-s6t8d\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:47.989863 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.989835    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume" pod "ffe43a31-90eb-4915-b944-733e2708f2e7" (UID: "ffe43a31-90eb-4915-b944-733e2708f2e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:33:47.991586 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:47.991559    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq" pod "ffe43a31-90eb-4915-b944-733e2708f2e7" (UID: "ffe43a31-90eb-4915-b944-733e2708f2e7"). InnerVolumeSpecName "kube-api-access-s4lsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:33:48.090402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.090315    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe43a31-90eb-4915-b944-733e2708f2e7-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:48.090402 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.090345    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4lsq\" (UniqueName: \"kubernetes.io/projected/ffe43a31-90eb-4915-b944-733e2708f2e7-kube-api-access-s4lsq\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:33:48.292591 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.292559    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"]
Nov 03 18:33:48.297340 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.297319    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d7aec-8c7c6df8cb4gqw"]
Nov 03 18:33:48.879347 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.879259    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68" event={"ID":"ffe43a31-90eb-4915-b944-733e2708f2e7","Type":"ContainerDied","Data":"3395bd05dc55f6d344a10da9ba5f53ef4566296e098d42990137c4edefc56943"}
Nov 03 18:33:48.879347 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.879325    6167 scope.go:122] "RemoveContainer" containerID="93b7d796acde53a668dc8b33e7dfe5f727438d92e897edcfb5098739a407d994"
Nov 03 18:33:48.879899 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.879264    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"
Nov 03 18:33:48.927985 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:48.927956    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" path="/var/lib/kubelet/pods/7a2856aa-00c0-45d8-b455-bf78132eb33e/volumes"
Nov 03 18:33:49.048272 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:49.048237    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"]
Nov 03 18:33:49.099946 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:49.099913    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d7aec-5c684q7w68"]
Nov 03 18:33:49.363769 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:49.363713    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:33:50.935494 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:50.935459    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" path="/var/lib/kubelet/pods/ffe43a31-90eb-4915-b944-733e2708f2e7/volumes"
Nov 03 18:33:52.203135 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:52.203079    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:33:52.203818 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:52.203774    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:33:54.521459 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:33:54.521382    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:33:55.106342 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:55.106303    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:55.119126 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:55.119104    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:33:55.956875 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:33:55.956845    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:34:24.520962 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:34:24.520922    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:34:30.061926 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:34:30.061887    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:34:48.139576 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:34:48.139542    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:34:51.471744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:34:51.470867    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:34:51.471744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:34:51.470956    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:34:54.521442 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:34:54.521395    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:34:59.592747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:34:59.592697    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"]
Nov 03 18:35:03.380845 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:03.380811    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:35:06.466890 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:06.466839    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-97fx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:35:06.467520 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:06.467486    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-97fx2" podUID="eabe656f-c8dd-4a46-9e7c-53694a6748ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:35:14.613063 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:14.613017    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58" podUID="cbdf801e-9d86-466c-8fa1-328870a44baf" containerName="main" containerID="cri-o://4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c" gracePeriod=45
Nov 03 18:35:14.870393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:14.870332    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:35:15.007052 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007022    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs\") pod \"cbdf801e-9d86-466c-8fa1-328870a44baf\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") "
Nov 03 18:35:15.007253 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007099    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm\") pod \"cbdf801e-9d86-466c-8fa1-328870a44baf\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") "
Nov 03 18:35:15.007253 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007196    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home\") pod \"cbdf801e-9d86-466c-8fa1-328870a44baf\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") "
Nov 03 18:35:15.007253 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007227    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6\" (UniqueName: \"kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6\") pod \"cbdf801e-9d86-466c-8fa1-328870a44baf\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") "
Nov 03 18:35:15.007514 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007314    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir\") pod \"cbdf801e-9d86-466c-8fa1-328870a44baf\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") "
Nov 03 18:35:15.007514 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007363    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache\") pod \"cbdf801e-9d86-466c-8fa1-328870a44baf\" (UID: \"cbdf801e-9d86-466c-8fa1-328870a44baf\") "
Nov 03 18:35:15.007917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007885    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache" pod "cbdf801e-9d86-466c-8fa1-328870a44baf" (UID: "cbdf801e-9d86-466c-8fa1-328870a44baf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:15.008074 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.007868    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir" pod "cbdf801e-9d86-466c-8fa1-328870a44baf" (UID: "cbdf801e-9d86-466c-8fa1-328870a44baf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:15.008154 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.008111    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home" pod "cbdf801e-9d86-466c-8fa1-328870a44baf" (UID: "cbdf801e-9d86-466c-8fa1-328870a44baf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:15.009537 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.009507    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm" pod "cbdf801e-9d86-466c-8fa1-328870a44baf" (UID: "cbdf801e-9d86-466c-8fa1-328870a44baf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:35:15.010842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.010817    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6" pod "cbdf801e-9d86-466c-8fa1-328870a44baf" (UID: "cbdf801e-9d86-466c-8fa1-328870a44baf"). InnerVolumeSpecName "kube-api-access-nwtz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:35:15.010951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.010848    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs" pod "cbdf801e-9d86-466c-8fa1-328870a44baf" (UID: "cbdf801e-9d86-466c-8fa1-328870a44baf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:35:15.109310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.109277    6167 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-tmp-dir\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:15.109310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.109304    6167 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-model-cache\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:15.109310 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.109313    6167 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdf801e-9d86-466c-8fa1-328870a44baf-tls-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:15.109555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.109321    6167 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-dshm\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:15.109555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.109330    6167 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdf801e-9d86-466c-8fa1-328870a44baf-home\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:15.109555 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.109339    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtz6\" (UniqueName: \"kubernetes.io/projected/cbdf801e-9d86-466c-8fa1-328870a44baf-kube-api-access-nwtz6\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:35:15.461772 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.461707    6167 generic.go:356] "Generic (PLEG): container finished" podID="cbdf801e-9d86-466c-8fa1-328870a44baf" containerID="4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c" exitCode=0
Nov 03 18:35:15.461964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.461793    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"
Nov 03 18:35:15.461964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.461748    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58" event={"ID":"cbdf801e-9d86-466c-8fa1-328870a44baf","Type":"ContainerDied","Data":"4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c"}
Nov 03 18:35:15.461964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.461867    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58" event={"ID":"cbdf801e-9d86-466c-8fa1-328870a44baf","Type":"ContainerDied","Data":"5e766fdded0e2c6c2095fe8048f797210b340bb2cfd45771e4208eb0541c98f3"}
Nov 03 18:35:15.461964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.461895    6167 scope.go:122] "RemoveContainer" containerID="4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c"
Nov 03 18:35:15.479663 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.479639    6167 scope.go:122] "RemoveContainer" containerID="4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c"
Nov 03 18:35:15.479957 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:35:15.479939    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c\": container with ID starting with 4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c not found: ID does not exist" containerID="4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c"
Nov 03 18:35:15.480010 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.479963    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c"} err="failed to get container status \"4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c\": rpc error: code = NotFound desc = could not find container \"4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c\": container with ID starting with 4238407f4e78ed5379dabbf918c98b6b55c33834f7c5b42973d0399cb334ad0c not found: ID does not exist"
Nov 03 18:35:15.653835 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.653805    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"]
Nov 03 18:35:15.705389 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:15.705357    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a3447aea-kserve-864884c7b9-btw58"]
Nov 03 18:35:16.927427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:16.927394    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdf801e-9d86-466c-8fa1-328870a44baf" path="/var/lib/kubelet/pods/cbdf801e-9d86-466c-8fa1-328870a44baf/volumes"
Nov 03 18:35:21.855405 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:21.855374    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:35:24.624830 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:35:24.615975    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:35:34.260245 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:34.260205    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:35:42.471053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:42.471014    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:35:45.449588 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:45.449562    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:35:45.463466 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:35:45.463442    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:35:54.520985 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:35:54.520947    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:36:05.083641 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.083602    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:36:05.704929 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.704895    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"]
Nov 03 18:36:05.705386 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705370    6167 cpu_manager.go:414] "containerMap: removing container" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerName="payload-processing"
Nov 03 18:36:05.705437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705388    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerName="payload-processing"
Nov 03 18:36:05.705437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705397    6167 cpu_manager.go:414] "containerMap: removing container" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerName="payload-pre-processing"
Nov 03 18:36:05.705437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705402    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerName="payload-pre-processing"
Nov 03 18:36:05.705437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705415    6167 cpu_manager.go:414] "containerMap: removing container" podUID="cbdf801e-9d86-466c-8fa1-328870a44baf" containerName="main"
Nov 03 18:36:05.705437 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705420    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdf801e-9d86-466c-8fa1-328870a44baf" containerName="main"
Nov 03 18:36:05.705592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705511    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2856aa-00c0-45d8-b455-bf78132eb33e" containerName="payload-processing"
Nov 03 18:36:05.705592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705519    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe43a31-90eb-4915-b944-733e2708f2e7" containerName="payload-pre-processing"
Nov 03 18:36:05.705592 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.705528    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdf801e-9d86-466c-8fa1-328870a44baf" containerName="main"
Nov 03 18:36:05.710020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.710002    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.785950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.785917    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"]
Nov 03 18:36:05.831986 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.831952    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d652d5d-6eae-42d9-88a4-dc18ee33311d-maas-api-tls\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.832144 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.832061    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d652d5d-6eae-42d9-88a4-dc18ee33311d-metrics-certs\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.832201 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.832142    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d652d5d-6eae-42d9-88a4-dc18ee33311d-openshift-service-ca\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.832249 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.832220    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srkf4\" (UniqueName: \"kubernetes.io/projected/5d652d5d-6eae-42d9-88a4-dc18ee33311d-kube-api-access-srkf4\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.932966 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.932938    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d652d5d-6eae-42d9-88a4-dc18ee33311d-openshift-service-ca\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.933131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.932992    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srkf4\" (UniqueName: \"kubernetes.io/projected/5d652d5d-6eae-42d9-88a4-dc18ee33311d-kube-api-access-srkf4\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.933131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.933021    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d652d5d-6eae-42d9-88a4-dc18ee33311d-maas-api-tls\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.933131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.933088    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d652d5d-6eae-42d9-88a4-dc18ee33311d-metrics-certs\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.933779 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.933749    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d652d5d-6eae-42d9-88a4-dc18ee33311d-openshift-service-ca\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.935481 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.935453    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d652d5d-6eae-42d9-88a4-dc18ee33311d-maas-api-tls\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.935481 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.935476    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d652d5d-6eae-42d9-88a4-dc18ee33311d-metrics-certs\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:05.995257 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:05.995188    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srkf4\" (UniqueName: \"kubernetes.io/projected/5d652d5d-6eae-42d9-88a4-dc18ee33311d-kube-api-access-srkf4\") pod \"maas-api-6d477cc994-5l2vf\" (UID: \"5d652d5d-6eae-42d9-88a4-dc18ee33311d\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:06.023075 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:06.023042    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:06.406952 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:36:06.406922    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d652d5d_6eae_42d9_88a4_dc18ee33311d.slice/crio-6f30d6564c4a6ee9d39d0b13fc0d99e43754faf05cc626efe31b7acfe0ae76fc WatchSource:0}: Error finding container 6f30d6564c4a6ee9d39d0b13fc0d99e43754faf05cc626efe31b7acfe0ae76fc: Status 404 returned error can't find the container with id 6f30d6564c4a6ee9d39d0b13fc0d99e43754faf05cc626efe31b7acfe0ae76fc
Nov 03 18:36:06.408574 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:06.408552    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:36:06.409854 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:06.409821    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"]
Nov 03 18:36:06.763656 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:06.763571    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf" event={"ID":"5d652d5d-6eae-42d9-88a4-dc18ee33311d","Type":"ContainerStarted","Data":"6f30d6564c4a6ee9d39d0b13fc0d99e43754faf05cc626efe31b7acfe0ae76fc"}
Nov 03 18:36:07.771477 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:07.771436    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf" event={"ID":"5d652d5d-6eae-42d9-88a4-dc18ee33311d","Type":"ContainerStarted","Data":"238999ccae77c31f8c6137030d4e0551c79c30186fcf0aa68ea05afa2e6ce4c5"}
Nov 03 18:36:07.771867 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:07.771501    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:07.899809 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:07.899759    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf" podStartSLOduration=2.04452576 podStartE2EDuration="2.899744641s" totalImagesPullingTime="855.218881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:07.898130973 +0000 UTC m=+3023.128456046" watchObservedRunningTime="2026-11-03 18:36:07.899744641 +0000 UTC m=+3023.130069713"
Nov 03 18:36:13.782530 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:13.782498    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d477cc994-5l2vf"
Nov 03 18:36:20.717291 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.717257    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"]
Nov 03 18:36:20.721179 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.721158    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.778077 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.778045    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-07d66926-serving-cert\""
Nov 03 18:36:20.831376 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.831346    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"]
Nov 03 18:36:20.882627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.882598    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.882778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.882659    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.882778 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.882739    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcvd\" (UniqueName: \"kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.882891 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.882878    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.984041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.983964    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.984041 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.984003    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.984231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.984044    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.984231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.984061    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcvd\" (UniqueName: \"kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.984792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.984767    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.986516 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.986486    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:20.986753 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:20.986707    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:21.046360 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:21.046330    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcvd\" (UniqueName: \"kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd\") pod \"maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:21.334969 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:21.334936    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:21.703217 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:36:21.703179    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3599c31_d602_48fd_9557_5175c0ed67d6.slice/crio-99720dcb7381ec10c368c3b27d59823fdd12e5c7d623a457afa6cf735d2a5589 WatchSource:0}: Error finding container 99720dcb7381ec10c368c3b27d59823fdd12e5c7d623a457afa6cf735d2a5589: Status 404 returned error can't find the container with id 99720dcb7381ec10c368c3b27d59823fdd12e5c7d623a457afa6cf735d2a5589
Nov 03 18:36:21.705456 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:21.705435    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"]
Nov 03 18:36:21.849182 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:21.849147    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt" event={"ID":"c3599c31-d602-48fd-9557-5175c0ed67d6","Type":"ContainerStarted","Data":"99720dcb7381ec10c368c3b27d59823fdd12e5c7d623a457afa6cf735d2a5589"}
Nov 03 18:36:22.615302 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:22.615262    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"]
Nov 03 18:36:22.615521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:22.615496    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878" podUID="3c90867b-73d5-46bf-bda6-5c71be0b9f7e" containerName="maas-api" containerID="cri-o://f1494c3c8fa66cb01437f06179decca635b1350476eff5398bfb52f1704f7cff" gracePeriod=30
Nov 03 18:36:22.856501 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:22.856418    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt" event={"ID":"c3599c31-d602-48fd-9557-5175c0ed67d6","Type":"ContainerStarted","Data":"cdd64bf74c81b91e58d810227c1aee2c89b24e338074629b674c59fa993c5569"}
Nov 03 18:36:22.856900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:22.856591    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:22.976816 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:22.976760    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt" podStartSLOduration=2.105245077 podStartE2EDuration="2.976747904s" totalImagesPullingTime="871.502827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:22.976391529 +0000 UTC m=+3038.206716613" watchObservedRunningTime="2026-11-03 18:36:22.976747904 +0000 UTC m=+3038.207072972"
Nov 03 18:36:23.866744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.866687    6167 generic.go:356] "Generic (PLEG): container finished" podID="3c90867b-73d5-46bf-bda6-5c71be0b9f7e" containerID="f1494c3c8fa66cb01437f06179decca635b1350476eff5398bfb52f1704f7cff" exitCode=0
Nov 03 18:36:23.867111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.866762    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878" event={"ID":"3c90867b-73d5-46bf-bda6-5c71be0b9f7e","Type":"ContainerDied","Data":"f1494c3c8fa66cb01437f06179decca635b1350476eff5398bfb52f1704f7cff"}
Nov 03 18:36:23.867111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.866813    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878" event={"ID":"3c90867b-73d5-46bf-bda6-5c71be0b9f7e","Type":"ContainerDied","Data":"e8cad1cc5987dd883f43c6f0010f1b6b5b4e28e78903467dd777c21ee1f4ca09"}
Nov 03 18:36:23.867111 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.866834    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8cad1cc5987dd883f43c6f0010f1b6b5b4e28e78903467dd777c21ee1f4ca09"
Nov 03 18:36:23.876390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.876364    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 18:36:23.914801 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.914769    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs\") pod \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") "
Nov 03 18:36:23.916960 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:23.916893    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs" pod "3c90867b-73d5-46bf-bda6-5c71be0b9f7e" (UID: "3c90867b-73d5-46bf-bda6-5c71be0b9f7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:36:24.016163 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.016121    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca\") pod \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") "
Nov 03 18:36:24.016489 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.016469    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5\" (UniqueName: \"kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5\") pod \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") "
Nov 03 18:36:24.016598 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.016555    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls\") pod \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\" (UID: \"3c90867b-73d5-46bf-bda6-5c71be0b9f7e\") "
Nov 03 18:36:24.016679 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.016639    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca" pod "3c90867b-73d5-46bf-bda6-5c71be0b9f7e" (UID: "3c90867b-73d5-46bf-bda6-5c71be0b9f7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:36:24.017010 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.016984    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:24.017127 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.017019    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:24.018967 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.018939    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5" pod "3c90867b-73d5-46bf-bda6-5c71be0b9f7e" (UID: "3c90867b-73d5-46bf-bda6-5c71be0b9f7e"). InnerVolumeSpecName "kube-api-access-cgtp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:36:24.019203 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.019185    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls" pod "3c90867b-73d5-46bf-bda6-5c71be0b9f7e" (UID: "3c90867b-73d5-46bf-bda6-5c71be0b9f7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:36:24.117657 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.117619    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtp5\" (UniqueName: \"kubernetes.io/projected/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-kube-api-access-cgtp5\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:24.117657 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.117653    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c90867b-73d5-46bf-bda6-5c71be0b9f7e-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:36:24.522321 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:36:24.522279    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:36:24.873169 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:24.873137    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"
Nov 03 18:36:25.044703 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:25.044671    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"]
Nov 03 18:36:25.096393 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:25.096364    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-w9878"]
Nov 03 18:36:26.927199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:26.927162    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c90867b-73d5-46bf-bda6-5c71be0b9f7e" path="/var/lib/kubelet/pods/3c90867b-73d5-46bf-bda6-5c71be0b9f7e/volumes"
Nov 03 18:36:28.873653 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:28.873623    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:36:29.146882 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:29.146804    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:36:46.214199 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:46.214162    6167 scope.go:122] "RemoveContainer" containerID="f1494c3c8fa66cb01437f06179decca635b1350476eff5398bfb52f1704f7cff"
Nov 03 18:36:54.520842 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:36:54.520761    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:36:59.798580 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:59.798543    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"]
Nov 03 18:36:59.799103 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:36:59.798771    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt" podUID="c3599c31-d602-48fd-9557-5175c0ed67d6" containerName="maas-api" containerID="cri-o://cdd64bf74c81b91e58d810227c1aee2c89b24e338074629b674c59fa993c5569" gracePeriod=30
Nov 03 18:37:00.067009 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.066971    6167 generic.go:356] "Generic (PLEG): container finished" podID="c3599c31-d602-48fd-9557-5175c0ed67d6" containerID="cdd64bf74c81b91e58d810227c1aee2c89b24e338074629b674c59fa993c5569" exitCode=0
Nov 03 18:37:00.067162 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.067040    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt" event={"ID":"c3599c31-d602-48fd-9557-5175c0ed67d6","Type":"ContainerDied","Data":"cdd64bf74c81b91e58d810227c1aee2c89b24e338074629b674c59fa993c5569"}
Nov 03 18:37:00.067244 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.067227    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt" event={"ID":"c3599c31-d602-48fd-9557-5175c0ed67d6","Type":"ContainerDied","Data":"99720dcb7381ec10c368c3b27d59823fdd12e5c7d623a457afa6cf735d2a5589"}
Nov 03 18:37:00.067307 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.067259    6167 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99720dcb7381ec10c368c3b27d59823fdd12e5c7d623a457afa6cf735d2a5589"
Nov 03 18:37:00.073709 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.073686    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:37:00.078827 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.078808    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs\") pod \"c3599c31-d602-48fd-9557-5175c0ed67d6\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") "
Nov 03 18:37:00.078936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.078847    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca\") pod \"c3599c31-d602-48fd-9557-5175c0ed67d6\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") "
Nov 03 18:37:00.078936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.078889    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls\") pod \"c3599c31-d602-48fd-9557-5175c0ed67d6\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") "
Nov 03 18:37:00.079014 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.078977    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd\" (UniqueName: \"kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd\") pod \"c3599c31-d602-48fd-9557-5175c0ed67d6\" (UID: \"c3599c31-d602-48fd-9557-5175c0ed67d6\") "
Nov 03 18:37:00.079240 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.079208    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca" pod "c3599c31-d602-48fd-9557-5175c0ed67d6" (UID: "c3599c31-d602-48fd-9557-5175c0ed67d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:37:00.079366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.079304    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3599c31-d602-48fd-9557-5175c0ed67d6-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:00.081038 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.081011    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd" pod "c3599c31-d602-48fd-9557-5175c0ed67d6" (UID: "c3599c31-d602-48fd-9557-5175c0ed67d6"). InnerVolumeSpecName "kube-api-access-9xcvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:00.081139 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.081037    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls" pod "c3599c31-d602-48fd-9557-5175c0ed67d6" (UID: "c3599c31-d602-48fd-9557-5175c0ed67d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:00.081139 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.081083    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs" pod "c3599c31-d602-48fd-9557-5175c0ed67d6" (UID: "c3599c31-d602-48fd-9557-5175c0ed67d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:00.180231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.180200    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:00.180231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.180228    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3599c31-d602-48fd-9557-5175c0ed67d6-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:00.180490 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:00.180240    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xcvd\" (UniqueName: \"kubernetes.io/projected/c3599c31-d602-48fd-9557-5175c0ed67d6-kube-api-access-9xcvd\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:37:01.072783 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:01.072754    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"
Nov 03 18:37:01.242390 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:01.242302    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"]
Nov 03 18:37:01.293412 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:01.293383    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-07d66926-5b875df7c5-w5kmt"]
Nov 03 18:37:02.927298 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:02.927265    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3599c31-d602-48fd-9557-5175c0ed67d6" path="/var/lib/kubelet/pods/c3599c31-d602-48fd-9557-5175c0ed67d6/volumes"
Nov 03 18:37:04.377079 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:04.377035    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:37:11.789776 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.789740    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"]
Nov 03 18:37:11.790423 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.790400    6167 cpu_manager.go:414] "containerMap: removing container" podUID="c3599c31-d602-48fd-9557-5175c0ed67d6" containerName="maas-api"
Nov 03 18:37:11.790521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.790425    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3599c31-d602-48fd-9557-5175c0ed67d6" containerName="maas-api"
Nov 03 18:37:11.790521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.790471    6167 cpu_manager.go:414] "containerMap: removing container" podUID="3c90867b-73d5-46bf-bda6-5c71be0b9f7e" containerName="maas-api"
Nov 03 18:37:11.790521 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.790479    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c90867b-73d5-46bf-bda6-5c71be0b9f7e" containerName="maas-api"
Nov 03 18:37:11.790674 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.790603    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c90867b-73d5-46bf-bda6-5c71be0b9f7e" containerName="maas-api"
Nov 03 18:37:11.790674 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.790622    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3599c31-d602-48fd-9557-5175c0ed67d6" containerName="maas-api"
Nov 03 18:37:11.795089 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.795069    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.849186 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.849157    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-2300af-openshift-default-dockercfg-xdvv4\""
Nov 03 18:37:11.894695 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.894661    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.894900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.894709    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.894900 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.894879    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.895029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.894926    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.895029 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.894979    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.895113 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.895086    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.895161 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.895126    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.895272 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.895252    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.895339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.895305    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kb5s\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.903535 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.903507    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"]
Nov 03 18:37:11.996524 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.996486    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.996684 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.996564    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kb5s\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.996684 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.996598    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.996814 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.996784    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.996925    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.996980    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997054    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997117    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997190    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997200    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997217    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997493    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.997601 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997538    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.998066 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.997892    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.999495 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.999471    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:11.999583 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:11.999517    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:12.059405 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.059339    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:12.059405 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.059389    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kb5s\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s\") pod \"e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") " pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:12.111699 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.111670    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:12.531639 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:37:12.531595    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832447df_dae3_45dc_aab3_040d66e17cac.slice/crio-557b8b85c30cd0fe3edccdf4396cf421f957ed9e0fe3daee9ab94061614356bc WatchSource:0}: Error finding container 557b8b85c30cd0fe3edccdf4396cf421f957ed9e0fe3daee9ab94061614356bc: Status 404 returned error can't find the container with id 557b8b85c30cd0fe3edccdf4396cf421f957ed9e0fe3daee9ab94061614356bc
Nov 03 18:37:12.538846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.537055    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"]
Nov 03 18:37:12.538846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.537433    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:37:12.538846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.537555    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:37:12.538846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:12.537635    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:37:13.141791 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:13.141749    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" event={"ID":"832447df-dae3-45dc-aab3-040d66e17cac","Type":"ContainerStarted","Data":"dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"}
Nov 03 18:37:13.142231 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:13.141819    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" event={"ID":"832447df-dae3-45dc-aab3-040d66e17cac","Type":"ContainerStarted","Data":"557b8b85c30cd0fe3edccdf4396cf421f957ed9e0fe3daee9ab94061614356bc"}
Nov 03 18:37:13.265670 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:13.265615    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podStartSLOduration=2.265601134 podStartE2EDuration="2.265601134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:13.264714389 +0000 UTC m=+3088.495039461" watchObservedRunningTime="2026-11-03 18:37:13.265601134 +0000 UTC m=+3088.495926200"
Nov 03 18:37:14.112091 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:14.112058    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:14.117624 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:14.117595    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:14.147455 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:14.147426    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:14.148287 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:14.148268    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:20.153171 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.153135    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"]
Nov 03 18:37:20.157303 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.157285    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.213810 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.213783    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"]
Nov 03 18:37:20.262064 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.262040    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-2300af-dockercfg-qv9dr\""
Nov 03 18:37:20.262064 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.262041    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-2300af\""
Nov 03 18:37:20.291923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.291898    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.292053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.291968    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxkr\" (UniqueName: \"kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr\") pod \"payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.393002 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.392958    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.393198 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.393054    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxkr\" (UniqueName: \"kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr\") pod \"payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.393682 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.393659    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.456053 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.455978    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxkr\" (UniqueName: \"kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr\") pod \"payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.470582 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.470557    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:20.839438 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:37:20.839404    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8c36e7_8ee1_478b_9a29_bcac5dd08b9f.slice/crio-ef15983bdd9bd12db84cbd7d8d85e6dc4bdad45d5e2c3f2322d04a448c0acb0a WatchSource:0}: Error finding container ef15983bdd9bd12db84cbd7d8d85e6dc4bdad45d5e2c3f2322d04a448c0acb0a: Status 404 returned error can't find the container with id ef15983bdd9bd12db84cbd7d8d85e6dc4bdad45d5e2c3f2322d04a448c0acb0a
Nov 03 18:37:20.842339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:20.842314    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"]
Nov 03 18:37:21.189831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:21.189745    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6" event={"ID":"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f","Type":"ContainerStarted","Data":"fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c"}
Nov 03 18:37:21.189831 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:21.189798    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6" event={"ID":"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f","Type":"ContainerStarted","Data":"ef15983bdd9bd12db84cbd7d8d85e6dc4bdad45d5e2c3f2322d04a448c0acb0a"}
Nov 03 18:37:21.190248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:21.189942    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:21.308763 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:21.308701    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6" podStartSLOduration=1.30868812 podStartE2EDuration="1.30868812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:21.307851856 +0000 UTC m=+3096.538176927" watchObservedRunningTime="2026-11-03 18:37:21.30868812 +0000 UTC m=+3096.539013191"
Nov 03 18:37:22.302887 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.302856    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"]
Nov 03 18:37:22.306691 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.306673    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.412472 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.412436    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-2300af-serving-cert\""
Nov 03 18:37:22.412969 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.412945    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8ct\" (UniqueName: \"kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.413084 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.412998    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.413128 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.413098    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.413167 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.413125    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.414728 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.414700    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"]
Nov 03 18:37:22.514339 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.514304    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8ct\" (UniqueName: \"kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.514549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.514356    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.514549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.514407    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.514549 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.514422    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.514549 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:37:22.514541    6167 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-serving-cert: secret "maas-api-e2e-authcfg-http-2300af-serving-cert" not found
Nov 03 18:37:22.514801 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:37:22.514612    6167 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls podName:5ef0f3a0-cebc-4595-a943-3cca09d8a0c7 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:23.014591204 +0000 UTC m=+3098.244916268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls") pod "maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" (UID: "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7") : secret "maas-api-e2e-authcfg-http-2300af-serving-cert" not found
Nov 03 18:37:22.515315 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.515285    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.517034 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.517007    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:22.579457 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:22.579427    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8ct\" (UniqueName: \"kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:23.020065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:23.019986    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:23.022537 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:23.022514    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:23.228951 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:23.228919    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:23.602005 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:37:23.601961    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef0f3a0_cebc_4595_a943_3cca09d8a0c7.slice/crio-2558d5b463f6c7a90d4ee0bb7840f17430a3789dcfd8006b9f20e08fcc47065d WatchSource:0}: Error finding container 2558d5b463f6c7a90d4ee0bb7840f17430a3789dcfd8006b9f20e08fcc47065d: Status 404 returned error can't find the container with id 2558d5b463f6c7a90d4ee0bb7840f17430a3789dcfd8006b9f20e08fcc47065d
Nov 03 18:37:23.605922 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:23.605893    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"]
Nov 03 18:37:24.207304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:24.207263    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" event={"ID":"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7","Type":"ContainerStarted","Data":"2558d5b463f6c7a90d4ee0bb7840f17430a3789dcfd8006b9f20e08fcc47065d"}
Nov 03 18:37:24.524435 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:37:24.524361    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:37:25.214466 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:25.214431    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" event={"ID":"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7","Type":"ContainerStarted","Data":"751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28"}
Nov 03 18:37:25.214867 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:25.214499    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:25.336893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:25.336837    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" podStartSLOduration=2.401681252 podStartE2EDuration="3.336818658s" totalImagesPullingTime="935.137406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:25.335495724 +0000 UTC m=+3100.565820795" watchObservedRunningTime="2026-11-03 18:37:25.336818658 +0000 UTC m=+3100.567143730"
Nov 03 18:37:31.226713 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:31.226680    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:37:31.525294 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:31.525206    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:37:32.199846 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:32.199818    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:37:41.332169 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:41.332136    6167 generic.go:356] "Generic (PLEG): container finished" podID="832447df-dae3-45dc-aab3-040d66e17cac" containerID="dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f" exitCode=0
Nov 03 18:37:41.332617 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:41.332206    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" event={"ID":"832447df-dae3-45dc-aab3-040d66e17cac","Type":"ContainerDied","Data":"dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"}
Nov 03 18:37:41.332617 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:41.332578    6167 scope.go:122] "RemoveContainer" containerID="dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"
Nov 03 18:37:41.333834 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:41.333808    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:37:41.333904 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:41.333873    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:37:41.333950 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:41.333913    6167 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 18:37:42.358971 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:42.358932    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" event={"ID":"832447df-dae3-45dc-aab3-040d66e17cac","Type":"ContainerStarted","Data":"a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893"}
Nov 03 18:37:43.111855 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:43.111820    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:43.112798 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:43.112773    6167 patch_prober.go:29] interesting pod/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:37:43.112923 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:43.112825    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 18:37:44.112131 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:44.112096    6167 patch_prober.go:29] interesting pod/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:37:44.112548 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:44.112155    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 18:37:44.152741 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:44.148854    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:45.112236 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:45.112199    6167 patch_prober.go:29] interesting pod/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:37:45.112600 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:45.112249    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 18:37:46.112733 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:46.112694    6167 patch_prober.go:29] interesting pod/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:37:46.113118 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:46.112781    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 18:37:47.112248 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:47.112215    6167 patch_prober.go:29] interesting pod/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 03 18:37:47.112426 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:47.112269    6167 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 03 18:37:48.116806 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:48.116769    6167 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:48.117824 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:37:48.117804    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:37:54.520774 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:37:54.520702    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:38:24.520613 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:38:24.520531    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:38:54.521328 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:38:54.521288    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:39:13.528833 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:13.528790    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"]
Nov 03 18:39:18.529454 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.529411    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6" podUID="0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" containerName="payload-pre-processing" containerID="cri-o://fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c" gracePeriod=25
Nov 03 18:39:18.781630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.781259    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:39:18.838690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.838658    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr\" (UniqueName: \"kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr\") pod \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") "
Nov 03 18:39:18.838690 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.838696    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume\") pod \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\" (UID: \"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f\") "
Nov 03 18:39:18.839164 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.839138    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume" pod "0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" (UID: "0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:18.840802 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.840781    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr" pod "0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" (UID: "0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f"). InnerVolumeSpecName "kube-api-access-4qxkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:18.914263 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.914234    6167 generic.go:356] "Generic (PLEG): container finished" podID="0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" containerID="fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c" exitCode=0
Nov 03 18:39:18.914427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.914294    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"
Nov 03 18:39:18.914427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.914322    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6" event={"ID":"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f","Type":"ContainerDied","Data":"fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c"}
Nov 03 18:39:18.914427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.914378    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6" event={"ID":"0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f","Type":"ContainerDied","Data":"ef15983bdd9bd12db84cbd7d8d85e6dc4bdad45d5e2c3f2322d04a448c0acb0a"}
Nov 03 18:39:18.914427 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.914403    6167 scope.go:122] "RemoveContainer" containerID="fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c"
Nov 03 18:39:18.925635 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.925617    6167 scope.go:122] "RemoveContainer" containerID="fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c"
Nov 03 18:39:18.925959 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:39:18.925935    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c\": container with ID starting with fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c not found: ID does not exist" containerID="fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c"
Nov 03 18:39:18.926065 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.925966    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c"} err="failed to get container status \"fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c\": rpc error: code = NotFound desc = could not find container \"fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c\": container with ID starting with fb1f4c36f6d43a6c3333d3871dd8adf4fe04593e34a6fddacc758b574777f67c not found: ID does not exist"
Nov 03 18:39:18.940110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.940081    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxkr\" (UniqueName: \"kubernetes.io/projected/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-kube-api-access-4qxkr\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:18.940110 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:18.940107    6167 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f-plugins-config-volume\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:19.085847 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:19.085821    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"]
Nov 03 18:39:19.091205 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:19.091183    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2300af-7fcbd75847-7vnw6"]
Nov 03 18:39:20.927627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:20.927594    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" path="/var/lib/kubelet/pods/0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f/volumes"
Nov 03 18:39:24.348462 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.348424    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"]
Nov 03 18:39:24.348943 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.348613    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" podUID="5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" containerName="maas-api" containerID="cri-o://751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28" gracePeriod=30
Nov 03 18:39:24.525042 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:39:24.524987    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:39:24.629578 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.629543    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:39:24.799821 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.799784    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\") pod \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") "
Nov 03 18:39:24.800017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.799918    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs\") pod \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") "
Nov 03 18:39:24.800017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.799948    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct\" (UniqueName: \"kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct\") pod \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") "
Nov 03 18:39:24.800017 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.799981    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca\") pod \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\" (UID: \"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7\") "
Nov 03 18:39:24.800419 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.800394    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca" pod "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" (UID: "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:24.802073 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.802046    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls" pod "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" (UID: "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:24.802175 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.802108    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct" pod "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" (UID: "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7"). InnerVolumeSpecName "kube-api-access-nt8ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:24.802175 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.802115    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs" pod "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" (UID: "5ef0f3a0-cebc-4595-a943-3cca09d8a0c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:24.901143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.901060    6167 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-maas-api-tls\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:24.901143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.901101    6167 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-metrics-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:24.901143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.901117    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8ct\" (UniqueName: \"kubernetes.io/projected/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-kube-api-access-nt8ct\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:24.901143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.901131    6167 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7-openshift-service-ca\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:24.957342 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.957309    6167 generic.go:356] "Generic (PLEG): container finished" podID="5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" containerID="751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28" exitCode=0
Nov 03 18:39:24.957493 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.957354    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"
Nov 03 18:39:24.957493 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.957364    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" event={"ID":"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7","Type":"ContainerDied","Data":"751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28"}
Nov 03 18:39:24.957493 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.957411    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4" event={"ID":"5ef0f3a0-cebc-4595-a943-3cca09d8a0c7","Type":"ContainerDied","Data":"2558d5b463f6c7a90d4ee0bb7840f17430a3789dcfd8006b9f20e08fcc47065d"}
Nov 03 18:39:24.957493 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.957442    6167 scope.go:122] "RemoveContainer" containerID="751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28"
Nov 03 18:39:24.970271 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.970246    6167 scope.go:122] "RemoveContainer" containerID="751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28"
Nov 03 18:39:24.970664 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:39:24.970634    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28\": container with ID starting with 751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28 not found: ID does not exist" containerID="751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28"
Nov 03 18:39:24.970803 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:24.970673    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28"} err="failed to get container status \"751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28\": rpc error: code = NotFound desc = could not find container \"751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28\": container with ID starting with 751bfea3a591fdefb7dfd9c6ac78507ff928a6d94bb0882a9336ec774462da28 not found: ID does not exist"
Nov 03 18:39:25.126143 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:25.126114    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"]
Nov 03 18:39:25.129917 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:25.129890    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2300af-7cd59bf6d8-tsqf4"]
Nov 03 18:39:26.927277 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:26.927245    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" path="/var/lib/kubelet/pods/5ef0f3a0-cebc-4595-a943-3cca09d8a0c7/volumes"
Nov 03 18:39:29.638187 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:29.638156    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:39:33.410924 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:33.410891    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"]
Nov 03 18:39:33.411428 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:33.411099    6167 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy" containerID="cri-o://a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893" gracePeriod=30
Nov 03 18:39:38.675957 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.675895    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:39:38.745391 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745361    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745544 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745416    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745544 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745439    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745544 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745540    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745568    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745610    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745633    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745859 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745679    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745859 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745705    6167 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s\") pod \"832447df-dae3-45dc-aab3-040d66e17cac\" (UID: \"832447df-dae3-45dc-aab3-040d66e17cac\") "
Nov 03 18:39:38.745977 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745863    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:38.745977 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.745957    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:38.746114 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.746089    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:38.746207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.746182    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:38.746256 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.746197    6167 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-data\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.746256 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.746245    6167 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-credential-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.746334 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.746264    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:38.747608 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.747585    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:38.747779 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.747759    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "kube-api-access-6kb5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:38.747964 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.747941    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:39:38.748047 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.747967    6167 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy" pod "832447df-dae3-45dc-aab3-040d66e17cac" (UID: "832447df-dae3-45dc-aab3-040d66e17cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:38.847436 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847404    6167 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-socket\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.847436 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847437    6167 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-workload-certs\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.847630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847451    6167 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832447df-dae3-45dc-aab3-040d66e17cac-istiod-ca-cert\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.847630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847465    6167 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kb5s\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-kube-api-access-6kb5s\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.847630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847480    6167 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832447df-dae3-45dc-aab3-040d66e17cac-istio-podinfo\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.847630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847492    6167 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832447df-dae3-45dc-aab3-040d66e17cac-istio-token\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:38.847630 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:38.847506    6167 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832447df-dae3-45dc-aab3-040d66e17cac-istio-envoy\") on node \"ip-10-0-141-108.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:39.040241 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.040208    6167 generic.go:356] "Generic (PLEG): container finished" podID="832447df-dae3-45dc-aab3-040d66e17cac" containerID="a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893" exitCode=0
Nov 03 18:39:39.040448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.040259    6167 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"
Nov 03 18:39:39.040448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.040273    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" event={"ID":"832447df-dae3-45dc-aab3-040d66e17cac","Type":"ContainerDied","Data":"a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893"}
Nov 03 18:39:39.040448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.040313    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg" event={"ID":"832447df-dae3-45dc-aab3-040d66e17cac","Type":"ContainerDied","Data":"557b8b85c30cd0fe3edccdf4396cf421f957ed9e0fe3daee9ab94061614356bc"}
Nov 03 18:39:39.040448 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.040333    6167 scope.go:122] "RemoveContainer" containerID="a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893"
Nov 03 18:39:39.050404 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.050385    6167 scope.go:122] "RemoveContainer" containerID="dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"
Nov 03 18:39:39.061020 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.060999    6167 scope.go:122] "RemoveContainer" containerID="a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893"
Nov 03 18:39:39.061288 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:39:39.061269    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893\": container with ID starting with a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893 not found: ID does not exist" containerID="a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893"
Nov 03 18:39:39.061358 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.061297    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893"} err="failed to get container status \"a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893\": rpc error: code = NotFound desc = could not find container \"a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893\": container with ID starting with a46ae194043497aa70a47e9eabecfe0d41277cda580f966495b8ae61b5a7c893 not found: ID does not exist"
Nov 03 18:39:39.061358 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.061313    6167 scope.go:122] "RemoveContainer" containerID="dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"
Nov 03 18:39:39.061558 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:39:39.061540    6167 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f\": container with ID starting with dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f not found: ID does not exist" containerID="dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"
Nov 03 18:39:39.061598 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.061562    6167 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f"} err="failed to get container status \"dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f\": rpc error: code = NotFound desc = could not find container \"dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f\": container with ID starting with dacec30c77ac0b1acecdba9e4ba36ccd44fffbc02cfcfd4914acaec0c5606b1f not found: ID does not exist"
Nov 03 18:39:39.209324 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.209237    6167 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"]
Nov 03 18:39:39.212932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.212896    6167 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2300af-openshift-default-798d8b95df-fr4bg"]
Nov 03 18:39:39.480905 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:39.480830    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:39:40.927260 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:40.927226    6167 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832447df-dae3-45dc-aab3-040d66e17cac" path="/var/lib/kubelet/pods/832447df-dae3-45dc-aab3-040d66e17cac/volumes"
Nov 03 18:39:49.780599 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:49.780563    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:39:52.209902 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:52.209846    6167 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-qp8zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:39:52.210304 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:39:52.209938    6167 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-qp8zz" podUID="63fe4ab5-a21e-4224-bc27-57fc19495f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:39:54.524772 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:39:54.524682    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:40:10.788314 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:10.788280    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:40:24.520885 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:40:24.520843    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:40:43.579757 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:43.579707    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g7dk4"]
Nov 03 18:40:45.528823 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:45.528790    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:40:45.553012 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:45.552983    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-w2mkr_1f389b73-c992-49ac-ba2d-cf69b91f2b1e/console-operator/1.log"
Nov 03 18:40:54.520895 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:40:54.520855    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:40:54.546216 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:54.546194    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wp8w9_416e0881-1aac-4e94-b0a9-25e4e6083aac/manager/0.log"
Nov 03 18:40:55.434327 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:55.434296    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c8b7c4cfc-vkfzr_91f13c90-2d9c-427a-b3e5-db90108c2284/manager/0.log"
Nov 03 18:40:56.326947 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:56.326911    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-txqgp_70cb4192-1966-47b3-b571-4bc47ed43919/manager/0.log"
Nov 03 18:40:57.399189 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:40:57.399150    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d477cc994-5l2vf_5d652d5d-6eae-42d9-88a4-dc18ee33311d/maas-api/0.log"
Nov 03 18:41:00.624773 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:00.624743    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kqnnj_00003c75-2730-4420-86db-53ebe728b1e5/manager/0.log"
Nov 03 18:41:00.951168 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:00.951080    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5xc7p_1d5a2954-b796-4c83-9725-8fd5085c87b1/manager/0.log"
Nov 03 18:41:01.413412 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:01.413384    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lqw2c_849ddfdc-3b2b-4c8d-852f-75849b3d3ef8/manager/0.log"
Nov 03 18:41:01.839250 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:01.839220    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-g7dk4_ad065c46-6333-426a-9269-4470118668f8/limitador/0.log"
Nov 03 18:41:02.066711 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:02.066680    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fss7p_f91e3668-e29d-438e-964a-1fe7eba2fe02/manager/0.log"
Nov 03 18:41:02.878030 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:02.877999    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs_49d03cb8-8b24-48e0-a8a8-befc390455ac/istio-proxy/0.log"
Nov 03 18:41:04.599059 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:04.599027    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-97fx2_eabe656f-c8dd-4a46-9e7c-53694a6748ca/istio-proxy/0.log"
Nov 03 18:41:04.887381 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:04.887293    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-9b9xq_bd508cab-c70c-4b66-8086-e582b8807ba6/istio-proxy/0.log"
Nov 03 18:41:05.912133 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:05.912098    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-qp8zz_63fe4ab5-a21e-4224-bc27-57fc19495f6e/istio-proxy/0.log"
Nov 03 18:41:06.884132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:06.884102    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7t5fd_12a4aea5-1d80-48ff-acdb-73d7404cee9f/payload-processing/0.log"
Nov 03 18:41:07.123531 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:07.123498    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b45cc6f6f-ph9d6_2947ed62-ecb7-4ccd-935b-b1870cffb3e5/router/0.log"
Nov 03 18:41:07.985329 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:07.985296    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8_3fe9fc29-00d4-46f5-ab5b-df8dd964a65c/main/0.log"
Nov 03 18:41:08.262492 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:08.262417    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m7nkv_6f01ae27-6219-4cf1-a0b7-5e1f190c2887/main/0.log"
Nov 03 18:41:08.549100 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:08.549011    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-pd4hh_8e691aa9-1efc-4031-b4c5-f0cfafe60438/main/0.log"
Nov 03 18:41:09.064993 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:09.064956    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj_4e1776c1-ff08-4072-bf71-83ed8d82beb1/main/0.log"
Nov 03 18:41:20.582634 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:20.582600    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wp8w9_416e0881-1aac-4e94-b0a9-25e4e6083aac/manager/0.log"
Nov 03 18:41:21.499623 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:21.499591    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c8b7c4cfc-vkfzr_91f13c90-2d9c-427a-b3e5-db90108c2284/manager/0.log"
Nov 03 18:41:22.168760 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:22.168731    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-txqgp_70cb4192-1966-47b3-b571-4bc47ed43919/manager/0.log"
Nov 03 18:41:23.075932 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:23.075864    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d477cc994-5l2vf_5d652d5d-6eae-42d9-88a4-dc18ee33311d/maas-api/0.log"
Nov 03 18:41:24.520749 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:41:24.520689    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:41:25.890842 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:25.890808    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kqnnj_00003c75-2730-4420-86db-53ebe728b1e5/manager/0.log"
Nov 03 18:41:26.213204 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:26.213130    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5xc7p_1d5a2954-b796-4c83-9725-8fd5085c87b1/manager/0.log"
Nov 03 18:41:26.680459 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:26.680426    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lqw2c_849ddfdc-3b2b-4c8d-852f-75849b3d3ef8/manager/0.log"
Nov 03 18:41:27.014593 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:27.014517    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-g7dk4_ad065c46-6333-426a-9269-4470118668f8/limitador/0.log"
Nov 03 18:41:27.244122 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:27.244094    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fss7p_f91e3668-e29d-438e-964a-1fe7eba2fe02/manager/0.log"
Nov 03 18:41:27.885627 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:27.885591    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs_49d03cb8-8b24-48e0-a8a8-befc390455ac/istio-proxy/0.log"
Nov 03 18:41:29.592255 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:29.592224    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-97fx2_eabe656f-c8dd-4a46-9e7c-53694a6748ca/istio-proxy/0.log"
Nov 03 18:41:29.866023 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:29.865942    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-9b9xq_bd508cab-c70c-4b66-8086-e582b8807ba6/istio-proxy/0.log"
Nov 03 18:41:30.874403 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:30.874366    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-qp8zz_63fe4ab5-a21e-4224-bc27-57fc19495f6e/istio-proxy/0.log"
Nov 03 18:41:31.808112 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:31.808079    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7t5fd_12a4aea5-1d80-48ff-acdb-73d7404cee9f/payload-processing/0.log"
Nov 03 18:41:32.040453 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:32.040421    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b45cc6f6f-ph9d6_2947ed62-ecb7-4ccd-935b-b1870cffb3e5/router/0.log"
Nov 03 18:41:32.715847 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:32.715818    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-j42j8_3fe9fc29-00d4-46f5-ab5b-df8dd964a65c/main/0.log"
Nov 03 18:41:33.001002 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:33.000922    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-m7nkv_6f01ae27-6219-4cf1-a0b7-5e1f190c2887/main/0.log"
Nov 03 18:41:33.273208 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:33.273121    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-pd4hh_8e691aa9-1efc-4031-b4c5-f0cfafe60438/main/0.log"
Nov 03 18:41:33.763204 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:41:33.763172    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rn5mj_4e1776c1-ff08-4072-bf71-83ed8d82beb1/main/0.log"
Nov 03 18:41:54.520945 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:41:54.520907    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:42:04.820278 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:04.820243    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c8b7c4cfc-vkfzr_91f13c90-2d9c-427a-b3e5-db90108c2284/manager/0.log"
Nov 03 18:42:05.415936 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:05.415908    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c8b7c4cfc-vkfzr_91f13c90-2d9c-427a-b3e5-db90108c2284/manager/0.log"
Nov 03 18:42:06.166681 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:06.166656    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7t5fd_12a4aea5-1d80-48ff-acdb-73d7404cee9f/payload-processing/0.log"
Nov 03 18:42:06.404348 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:06.404322    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7t5fd_12a4aea5-1d80-48ff-acdb-73d7404cee9f/payload-processing/0.log"
Nov 03 18:42:24.520696 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:42:24.520646    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:42:44.773058 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:44.773027    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c8b7c4cfc-vkfzr_91f13c90-2d9c-427a-b3e5-db90108c2284/manager/0.log"
Nov 03 18:42:45.366871 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:45.366835    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c8b7c4cfc-vkfzr_91f13c90-2d9c-427a-b3e5-db90108c2284/manager/0.log"
Nov 03 18:42:46.110207 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.110180    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7t5fd_12a4aea5-1d80-48ff-acdb-73d7404cee9f/payload-processing/0.log"
Nov 03 18:42:46.307792 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.307764    6167 scope.go:122] "RemoveContainer" containerID="cdd64bf74c81b91e58d810227c1aee2c89b24e338074629b674c59fa993c5569"
Nov 03 18:42:46.337289 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.337257    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-7t5fd_12a4aea5-1d80-48ff-acdb-73d7404cee9f/payload-processing/0.log"
Nov 03 18:42:46.689623 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.689594    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8k5h/must-gather-xsgjg"]
Nov 03 18:42:46.690079 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690065    6167 cpu_manager.go:414] "containerMap: removing container" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy"
Nov 03 18:42:46.690132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690081    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy"
Nov 03 18:42:46.690132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690093    6167 cpu_manager.go:414] "containerMap: removing container" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy"
Nov 03 18:42:46.690132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690098    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy"
Nov 03 18:42:46.690132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690119    6167 cpu_manager.go:414] "containerMap: removing container" podUID="5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" containerName="maas-api"
Nov 03 18:42:46.690132 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690124    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" containerName="maas-api"
Nov 03 18:42:46.690292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690135    6167 cpu_manager.go:414] "containerMap: removing container" podUID="0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" containerName="payload-pre-processing"
Nov 03 18:42:46.690292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690140    6167 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" containerName="payload-pre-processing"
Nov 03 18:42:46.690292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690212    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy"
Nov 03 18:42:46.690292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690222    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="832447df-dae3-45dc-aab3-040d66e17cac" containerName="istio-proxy"
Nov 03 18:42:46.690292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690231    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef0f3a0-cebc-4595-a943-3cca09d8a0c7" containerName="maas-api"
Nov 03 18:42:46.690292 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.690238    6167 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8c36e7-8ee1-478b-9a29-bcac5dd08b9f" containerName="payload-pre-processing"
Nov 03 18:42:46.700208 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.700183    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:46.746672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.746636    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78j2v\" (UniqueName: \"kubernetes.io/projected/07281f5c-a1f4-488b-894a-cbfc1c80b972-kube-api-access-78j2v\") pod \"must-gather-xsgjg\" (UID: \"07281f5c-a1f4-488b-894a-cbfc1c80b972\") " pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:46.746672 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.746681    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07281f5c-a1f4-488b-894a-cbfc1c80b972-must-gather-output\") pod \"must-gather-xsgjg\" (UID: \"07281f5c-a1f4-488b-894a-cbfc1c80b972\") " pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:46.756763 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.756712    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8k5h\"/\"openshift-service-ca.crt\""
Nov 03 18:42:46.760112 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.760028    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/must-gather-xsgjg"]
Nov 03 18:42:46.808137 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.808113    6167 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8k5h\"/\"kube-root-ca.crt\""
Nov 03 18:42:46.808283 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.808212    6167 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k8k5h\"/\"default-dockercfg-dhhjm\""
Nov 03 18:42:46.847852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.847809    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78j2v\" (UniqueName: \"kubernetes.io/projected/07281f5c-a1f4-488b-894a-cbfc1c80b972-kube-api-access-78j2v\") pod \"must-gather-xsgjg\" (UID: \"07281f5c-a1f4-488b-894a-cbfc1c80b972\") " pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:46.847852 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.847860    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07281f5c-a1f4-488b-894a-cbfc1c80b972-must-gather-output\") pod \"must-gather-xsgjg\" (UID: \"07281f5c-a1f4-488b-894a-cbfc1c80b972\") " pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:46.848171 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.848154    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07281f5c-a1f4-488b-894a-cbfc1c80b972-must-gather-output\") pod \"must-gather-xsgjg\" (UID: \"07281f5c-a1f4-488b-894a-cbfc1c80b972\") " pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:46.910084 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:46.910053    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78j2v\" (UniqueName: \"kubernetes.io/projected/07281f5c-a1f4-488b-894a-cbfc1c80b972-kube-api-access-78j2v\") pod \"must-gather-xsgjg\" (UID: \"07281f5c-a1f4-488b-894a-cbfc1c80b972\") " pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:47.029287 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:47.029208    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/must-gather-xsgjg"
Nov 03 18:42:47.394948 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:42:47.394918    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07281f5c_a1f4_488b_894a_cbfc1c80b972.slice/crio-6875e1b986bba763d5e2c2d4e42d58cffc020876ef257b5e4103ad0e3264763c WatchSource:0}: Error finding container 6875e1b986bba763d5e2c2d4e42d58cffc020876ef257b5e4103ad0e3264763c: Status 404 returned error can't find the container with id 6875e1b986bba763d5e2c2d4e42d58cffc020876ef257b5e4103ad0e3264763c
Nov 03 18:42:47.396941 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:47.396921    6167 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:42:47.398035 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:47.398012    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/must-gather-xsgjg"]
Nov 03 18:42:48.139447 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:48.139407    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/must-gather-xsgjg" event={"ID":"07281f5c-a1f4-488b-894a-cbfc1c80b972","Type":"ContainerStarted","Data":"6875e1b986bba763d5e2c2d4e42d58cffc020876ef257b5e4103ad0e3264763c"}
Nov 03 18:42:49.146688 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:49.146655    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/must-gather-xsgjg" event={"ID":"07281f5c-a1f4-488b-894a-cbfc1c80b972","Type":"ContainerStarted","Data":"8c2337b99713989d4d2c1a570ec87ef29aa3e695c204cdad898a492b8ae53c6a"}
Nov 03 18:42:49.147218 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:49.146696    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/must-gather-xsgjg" event={"ID":"07281f5c-a1f4-488b-894a-cbfc1c80b972","Type":"ContainerStarted","Data":"cd47bd9763e48108b5fdf32ec7e8834b6f5e7d60f192c98d81b2a4ba3b5bb4eb"}
Nov 03 18:42:49.265149 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:49.265090    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k8k5h/must-gather-xsgjg" podStartSLOduration=1.889625118 podStartE2EDuration="3.26507706s" totalImagesPullingTime="1.375451942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:42:49.264221865 +0000 UTC m=+3424.494546937" watchObservedRunningTime="2026-11-03 18:42:49.26507706 +0000 UTC m=+3424.495402125"
Nov 03 18:42:54.521537 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:42:54.521460    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:42:56.974731 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:56.973265    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zc2rm_9807c6c6-ef8b-4769-bf2e-97b5c5a23604/global-pull-secret-syncer/0.log"
Nov 03 18:42:57.593752 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:57.590108    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-65f6r_9392d9f2-6a5c-4b6a-b048-a20227a79836/connection-checker/0.log"
Nov 03 18:42:58.036456 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:58.036374    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7dsfq_8110d47e-1f9c-480d-a1b6-d4a2b2b1e911/connection-checker/0.log"
Nov 03 18:42:58.505744 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:58.503244    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-txjqv_0d4e187c-93a8-4dc1-bd70-4c4dfc3b2fb5/connection-checker/0.log"
Nov 03 18:42:59.973399 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:42:59.973358    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j2vd6_50141ec0-76c0-453c-9ecc-e5c7c227dad2/konnectivity-agent/0.log"
Nov 03 18:43:01.223442 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:01.223390    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-108.us-west-2.compute.internal_acca05a04d25dcd6bb0ab5ae22e6769a/haproxy/0.log"
Nov 03 18:43:05.080355 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.080321    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kqnnj_00003c75-2730-4420-86db-53ebe728b1e5/manager/0.log"
Nov 03 18:43:05.488275 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.488199    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5xc7p_1d5a2954-b796-4c83-9725-8fd5085c87b1/manager/0.log"
Nov 03 18:43:05.577232 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.577197    6167 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"]
Nov 03 18:43:05.584670 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.584645    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.639366 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.639327    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"]
Nov 03 18:43:05.690031 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.689988    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-proc\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.690480 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.690436    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-podres\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.690618 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.690549    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhlw\" (UniqueName: \"kubernetes.io/projected/27071b5b-8bcf-4f1b-969b-08174a37dbfd-kube-api-access-fhhlw\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.690700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.690646    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-lib-modules\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.690700 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.690686    6167 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-sys\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791570    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-lib-modules\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791660 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791612    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-sys\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791705    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-lib-modules\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791767    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-proc\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791778    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-sys\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791789    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-podres\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791838    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-podres\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791849    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27071b5b-8bcf-4f1b-969b-08174a37dbfd-proc\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.791933 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.791869    6167 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhlw\" (UniqueName: \"kubernetes.io/projected/27071b5b-8bcf-4f1b-969b-08174a37dbfd-kube-api-access-fhhlw\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.853832 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.853803    6167 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhlw\" (UniqueName: \"kubernetes.io/projected/27071b5b-8bcf-4f1b-969b-08174a37dbfd-kube-api-access-fhhlw\") pod \"perf-node-gather-daemonset-qn7gp\" (UID: \"27071b5b-8bcf-4f1b-969b-08174a37dbfd\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:05.899893 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:05.899862    6167 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:06.275761 ip-10-0-141-108 kubenswrapper[6167]: W1103 18:43:06.275698    6167 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27071b5b_8bcf_4f1b_969b_08174a37dbfd.slice/crio-16f7d31b6daa59700fb6f1f8937b66e4adc8f07a4898f42d0b20ee070ebb4f41 WatchSource:0}: Error finding container 16f7d31b6daa59700fb6f1f8937b66e4adc8f07a4898f42d0b20ee070ebb4f41: Status 404 returned error can't find the container with id 16f7d31b6daa59700fb6f1f8937b66e4adc8f07a4898f42d0b20ee070ebb4f41
Nov 03 18:43:06.278570 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:06.278548    6167 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"]
Nov 03 18:43:06.300234 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:06.300202    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp" event={"ID":"27071b5b-8bcf-4f1b-969b-08174a37dbfd","Type":"ContainerStarted","Data":"16f7d31b6daa59700fb6f1f8937b66e4adc8f07a4898f42d0b20ee070ebb4f41"}
Nov 03 18:43:06.330747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:06.330690    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lqw2c_849ddfdc-3b2b-4c8d-852f-75849b3d3ef8/manager/0.log"
Nov 03 18:43:06.716966 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:06.716933    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-g7dk4_ad065c46-6333-426a-9269-4470118668f8/limitador/0.log"
Nov 03 18:43:07.183276 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:07.183245    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fss7p_f91e3668-e29d-438e-964a-1fe7eba2fe02/manager/0.log"
Nov 03 18:43:07.307733 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:07.307688    6167 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp" event={"ID":"27071b5b-8bcf-4f1b-969b-08174a37dbfd","Type":"ContainerStarted","Data":"d957264a400efa9f22b3fab9a35aa804783c3365dce324a5bfdd3a845ce70183"}
Nov 03 18:43:07.308141 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:07.307778    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:07.426746 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:07.426685    6167 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp" podStartSLOduration=2.426672418 podStartE2EDuration="2.426672418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:07.425157249 +0000 UTC m=+3442.655482319" watchObservedRunningTime="2026-11-03 18:43:07.426672418 +0000 UTC m=+3442.656997490"
Nov 03 18:43:12.360246 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:12.360214    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8ff8h_69cbe72d-a018-4a89-8951-a25e50d6d33b/cluster-monitoring-operator/0.log"
Nov 03 18:43:13.325108 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:13.325082    6167 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-qn7gp"
Nov 03 18:43:14.726153 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:14.726121    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6gk44_33b55ac5-4039-4da9-81f2-b64fadc22435/node-exporter/0.log"
Nov 03 18:43:15.130676 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:15.130646    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6gk44_33b55ac5-4039-4da9-81f2-b64fadc22435/kube-rbac-proxy/0.log"
Nov 03 18:43:15.535747 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:15.535670    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6gk44_33b55ac5-4039-4da9-81f2-b64fadc22435/init-textfile/0.log"
Nov 03 18:43:23.090228 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:23.090194    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-v9k7l_36506792-bfe4-4a9f-b27c-c2096cb67ebe/prometheus-operator-admission-webhook/0.log"
Nov 03 18:43:24.522132 ip-10-0-141-108 kubenswrapper[6167]: E1103 18:43:24.522096    6167 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hprg4" containerName="extractor"
Nov 03 18:43:28.666338 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:28.666301    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cmwnjs_49d03cb8-8b24-48e0-a8a8-befc390455ac/istio-proxy/0.log"
Nov 03 18:43:29.401305 ip-10-0-141-108 kubenswrapper[6167]: I1103 18:43:29.401265    6167 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-kkbq7_08a559aa-cebd-40a6-9393-3feb287029ee/networking-console-plugin/0.log"